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PREFACE 



As noted in the text, this document is one of number of reports 
resulting from the evaluation of the impact of educational research 
and development products. Available separately are the 21 Product 
Development Reports listed in Appendix H. Each Product Development 
Report represents a relatively unique attempt to document what 
occurred in the development of a recent educational product that 
appeared to have impact in the nation’s schools. Although awareness 
of the full extent of this study requires reading both the individual 
Product Development Reports and this final report, each may be read 
individually. 

A number of AIR staff have participated in the study. The per- 
manent project staff, in AIR's Evaluations and Research Program, 
consisted of: Calvin E. Wright, Principal Investigator; Jack J. 

Crawford, Project Director; Daniel W. Kratochvil, Associate Project 
Director; Carolyn A. Morrow, Administrative Assistant; and Sibyl 0. 
Anderson, Administrative Associate. Other staff who assisted in the 
initial planning activities, site visits, or preparation of the Pro- 
duct Development Reports include: Joan M. Altick; Bonita J. Berger; 

Steven M. Jung; Yungho Kim; Dewey Lipe; William L. Raley; Barbara A. 
Sanderson; Lorna J. Thompson; David V. Tiedeman (whose name was in- 
advertently omitted as a coauthor of Product Development Report No. 
11: The Arithmetic Proficiency Training Program); and Robert A. 

Weisgerber. Both staff and project owe much to the perceptive 
criticism of Dr. Alice Y. Scates, Project Monitor for the Office of 
Program; Planning and Evaluation. 
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INTRODUCTION 



Problem 



According to the 1969 USOE publication entitled Educational 
Research and Development in the United States , Office of Education 
funds spent in support of educational product development since 1957 
amount to a total of over $2 billion.^ Overall U.S. government sup- 
port for educational products, roughly defined, during the fiscal 
year 1968 was over $91 million. This amount does not include OEO 
expenditures in the area of instructional system development , which 
would substantially Increase the figure. 

Various attempts to measure the results of this investment have 
been made, such as the works of Getzels (1969) and Cronbach and Suppes 
(1969) . In general, these reports tend to show how significant lines 
of conclusion-oriented basic inquiry have borne fruit in the overall 
context of educational practice. It is apparent, ..owever, that most 
evaluation activities which are connected with the educational research 
and development process fall under the rubric of decision-oriented or 
applied inquiry. The nature of these activities, directed primarily 
toward revising an educational product, makes it extremely difficult 
to obtain a precise reading of the overall impact of the product. 

Product impact may be usefully thought of as measurable positive change 
in student performance on educationally important goals sustained over 
time. There is an extreme need for valid Indicators of such Impact. 

The primary problem, then, is to begin development of a set of 
procedures whereby specific evidence of potential product Impact may 
be Identified so that an assessment of R&D impact can be made. These 
procedures could be utilized in future decisions regarding the direc- 
tion of and support for educational research and development efforts 
undertaken by USOE. For example, such findings as performance in 
pilot study or field test situations must be considered as strong 
Indicators of the potential of a product to ultimately produce measur- 
able change In student performance. 

The case for using empirical findings from decision-oriented 
product development research is strengthened when one considers the 
results of the special 1968-69 study for the Organization for Economic 
Cooperation and Development commissioned through the Policy Institute 
of the Syracuse University Research Corporation (USOE, 1969). One of 
the most interesting outcomes of this study was that 64% of the school 
superintendents in a national sample gave no response to a question 
asking for the educational research and development outcomes which 
have had an Influence on American school practice. A secondary problem, 
then, ic to provide Information which will allow USOE to effectively 



^ This includes listed expenditures in the areas of instructional 
systems, curriculum, computer -managed or assisted instruction, and a 
combination category which contains the regional laboratory expenditures. 
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coordinate the dissemination of products which have identified potential 
for educational Impact . 



Review of Literature and Relevant Research 

The central purposes of the U.S. Office of Education research pro- 
gram have been stated as: the generation of knowledge; the development 

of validated and economically feasible instructional products; and the 
dissemination of information that will enable local schools to be aware 
of and Implement the new knowledge and techniques (USOE, 1968). 

Various studies have been conducted to explain both the process of 
relating new knowledge to the development of validated educational pro- 
ducts, and the process of insuring that validated products eventually 
come to have an impact on local school practices. Several examples of 
both processes will be noted here. 

Of historical and methodological significance are studies such as 
the Department of Defense Project HINDSIGHT (Sherwin and Isensen , 1966) , 
the National Science Foundation Project TRACES (1968), and the National 
Academy of Sciences report to the House Committee on Science and Astro- 
nautics (1967). These studies have sought to identify the principal ele- 
ments of successful applied research leading to new technology of national 
importance. All of the reports contain detailed examples of the develop- 
ment of exemplary products and processes. In addition, these studies 
have Indicated certain characteristics of organizations, environments, 
and personnel which seemed to be indicators of successful research and 
development projects, and Include proposals for more effective applica- 
tion of scientific resources to the solution of national problems. For 
example, one such project concluded: 

1. In all cases studied, nonmlsslon research provided the 
origins from which science and technology could advance 
toward the innovations which lay ahead. 

2. Of the key events documented, approximately 70% were 
nonmlsslon research, 20% mission-oriented research, 
and 10% development and application. 

3. The number of nonmlsslon events peaks significantly 
between the 20th and 30th year prior to an Innovation, 
while mission-oriented research events and those in 
the development and application area peak during the 
decade preceding innovation. 

4. For the cases studied the average time from the concep- 
tion to demonstration of an innovation was nine years. 

5. Most nonmlsslon research Is completed prior to the 
conception of the Innovation to which it will ulti- 
mately contribute. 

6. Although nonmlsslon and mlsslon-orlented activities - 
regress during the several years just preceding 



innovation, it is apparent that the interplay between 
these types of research activities is important and 
sometimes even crucial during this terminal period. 

7. The presence of interdisciplinary communication is 
very evident in, and important to, the achievement 
of innovation. 

8. The study points out the need for a better under- 
standing concerning the two-way interaction between 
science and technology. The tracings revealed cases 
in which mission-oriented research or development 
efforts elicited later nonmission research which often 
was found to be crucial to the ultimate innovation. 

9. Innovations for the next generations depend on today’s 
nonmission research. [NSF, 1968, pp. iv-v.] 

Against this prologue, it is interesting to note some of the find- 
ings of the Committee on Educational Research of the National Academy of 
Education edited by Lee Cronbach and Patrick Suppes (1969). A similar 
approach is followed, in that the committee sought to point out current 
practical manifestations of exemplary lines of conclusion-oriented 
(basic) inquiry and then went on to formulate a position whereby both 
research and development can be better integrated into the stream of 
educational progress. This position is summarized as follows: 

1) The main function of conclusion-oriented inquiry 
is to gain a better understanding of some educational pro- 
cess, or of some aspect of the nature of man and society. 

The research that has the greatest long-term significance is 
that which produces new concepts, rather than that which 
attempts to use the delicate apparatus of science to hammer 
out an Immediate solution to a current problem. 

2) An attempt to solve an immediate problem can also 
capitalize on inquiry, first by using the available knowledge 
and concepts to work out a sensible plan, second by usi ^ 
direct decision-oriented inquiry to guide day-to-day cr.c.' ;?ions 
as the plan is put into practice, and also to determine how 
well the final scheme works. 

3) The two kinds of inquiry which we have referred to 
as "conclusion-oriented" and "decision-oriented" are carried 
out with different time perspectives and often require differ- 
ent techniques. Hence it is important to distinguish the two 
functions, and to call upon each type of inquiry for the kind 
of service it can best render. Decision-oriented studies are 
studies in the service of a decision-maker, and they will be 
useful only if the decision-maker truly understands what they 
can contribute and demands that kind of help. Conclusion- 
oriented studies are generalized, their broad conclusions and 
new ways of thinking are intellectual resources for educators, 
scholars, and laymen, but they do not give a pinpointed answer 
to the problems the decision-maker faces today. 



4) It is a mistake to press the conclusion-oriented 
inquirer to arrive at generalizations that the practical 
educator "can apply." Even in highly technological fields, 
basic understandings gained through fundamental research are 
not directly applied; there must be additional conclusion- 
oriented research to understand the practical problem, and 
then there must be decision-oriented design studies and try- 
outs to produce a practical solution for a particular situa- 
tion. Enlightenment gained from conclusion-oriented studies 
helps the designer to know what to take into account in his 
plans and his evaluations, but it never dictates the shape 
of the practical solution. [Cronback and Suppes, 1969, 

pp. 266—268]. 

Ini the area of dissemination and adoption of educational research 
and development products, several extensive reviews of the available 
literature and several comprehensive theoretical models have recently 
appeared. The report of the Commission for Economic Development has 
stated the problem "... but unless the products of research are 
disseminated effectively in forms to be usefully employed, the invest- 
ment is lost" (CED, 1968, p. 29). The authors of the 1969 report 
on Educational Research and Development in the United States (USOE, 

1969) •( ®*tudying in depth the extent of usage of various educational 
res^i^sWK' development products, concluded that "the overwhelming 

'(^ladents get no exposure to most of the newer teaching 
praetift^. . . . More than half of the 33,731,000 students included 
in ouf .projection got no exposure to 13 of the 17 specified innova- 
tions (in the project's questionnaire]" (p. 147). 

Again it is apparent that the dissemination cycle in the area of 
the physical sciences may provide a model to education. Phillips (1969) 
outlines the system through which technology developed by the NASA 
manned space flight program has found wide application in the bio- 
medical field. In comparison, models of dissemination and change in 
education are quite rough. The Clark and Cuba research, development, 
and dissemination paradigm (1965) seems to underlie much of the cur- 
rent policy planning at USOE. However, the model has been criticised 
on several accounts. For example, Gideonse (1968) points out its 
failure to account for initiatives of different kinds which may take 
place at any point on the continuum from research through adoption. 
Essentially this is a rejection of the "linear" dissemination and 
utilization model. Kreitlow and MacNeil (1969) among others have 
proposed a cyclic model for which they have collected some supporting 
data. 



Perhaps the most well-known and well-supported current dissemina- 
tion model is that of Havelock (1969) who has examined the literature 
on utilization and adoption at length. He and his associates have 
proposed a "linkage" model which is a combination of several other 
paradigms. .With regard to this model, it is suggested that effec- 
tive knowledge utilization requires a degree of division of labor , 
coordination, and collaboration throughout the social system. The 



role of government should be to • . . support, facilitate and 

coordinate linkage activities so that the total system can function 
more effectively" (Havelock, 1969). By way of suggesting ways to 
collect data supporting this paradigm, Havelock suggested that 

. . although there were many quantitative research studies, there 
was a paucity of case materials . We need more case studies which 
carefully document and report dissemination and utilization events* 
(Havelock, 1969, p. 11-2). The research proposed here bears upon 
this need. 



Objectives 

The overall goal of this exploratory project was to Initially 
uncover factors or variables likely to be Involved In the Impact of 
educational products. Hopefully, the results of the search would 
then facilitate the eventual development of a more precise system 
of gauging product Impact. Such a system could assist USOE In 
deciding how to Invest R&D funds for product development to obtain 
maximum Impact on our educational system. 

The specific objectives of this exploratory study of educational 
products were: 

1. Primarily, to construct systematic case studies of the 
development of selected educational products as an Initial 
basis for arriving at more precise procedures to assess 
potential Impact. 

2. Secondarily, to provide empirical case study data rele- 
vant to current hypotheses about Innovation and change 
In public school education. 

3. An additional and enabling objective was to prepare a 
listing of educational products judged, according to 
specific criteria, as having substantial positive Impact. 



Approach 

This study utilized an approach similar to those which have been 
used successfully In the physical sciences and which have been advo- 
cated by authors In the field of knowledge dissemination and utiliza- 
tion. In effect , this Is to carefully identify products which have 
"made It," that Is, have emerged from the research and development 
cycle to Impact with success upon American education. In-depth study 
of the educational products was undertaken to provide a basis for 
establishing what features of the products— their origins, their 
developers, their target populations, their dissemination, the pre- 
vailing Zeitgeist , etc.— might be identified as common contributors 
to their eventual impact. Generalizations arising from such study 
can be used, to construct Indicators of potential product Impact, and 
to provide evidence relevant to various models of the research. 



dissemination, and development process. In one sense, then, this 
study has both applied and basic outcomes. 

The study included the following activities : 

1. The development of initial selection criteria for 
educational products with impact. 

2. The identification and description of products meeting 
the initial selection criteria. 

3. The Identification of approximately 20 of those pro- 
ducts for intensive review. 

4. Systematic case studies of each product tracing the 
history of each of the 20 products from its origin to 
its diffusion and adoption. 

5. The extraction of generalizations about the products 
based upon product characteristics, development, 
evaluation, diffusion, adoption requirements, etc. 

The generalizations were then related to existing 
hypotheses regarding the processes of Innovation and 
change in education. 

6. An identification of Ways whereby information on the 
Impact, or potential impact, of educational products 
may be obtained. 
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METHOD 



Development of Initial Selection Criteria 

Specific criteria were developed ' for the selection of educational 
research and development products that have had substantial positive 
Impact. It was recognized that It is difficult to measure educational 
Impact, which might be defined as sustained, measurable, and desirable 
change In educational achievement or the factors which directly deter- 
mine It. Since change In educational achievement Is also difficult 
to measure, one possibility would have been to examine the factors or 
variables likely to be Involved In determining It. It was believed, 
however, that a crucial criterion for determining educational Impact 
Is a demonstrated effectiveness In Improving student performance on 
well-recognized cognitive and/or affective goals. Previous work at 
the American Institutes for Research (AIR) by Hawkrldge et al (1968; 
1969), had demonstrated the feasibility of Including In studies of 
exemplary programs only those that have demonstrated statistically 
significant gains with reference to national norms on standardized 
tests, or significant gains for enrolled students over control students. 
It was, therefore, decided that such considerations be Included In the 
criteria for the Initial selection of educational products for this 
study. 

It was recognized that restricting the consideration of educa- 
tional products only to those that have. In fact, demonstrated effec- 
tiveness In Improving student performance might result In overly 
restricting the n umb er of products to be considered when simultaneously 
applied with other criteria covering potential antecedents of demon- 
strated effectiveness. It was decided, therefore, that demonstrated 
effectiveness would be considered separately and In addition to other 
criteria, such as scope of use. Definitions of Impact In areas other 
than educational achievement were also considered. Preliminary work 
at the American Institutes for Research had, for example, identified 
some eleven goals considered essential for students In today's schools 
(Wright, 1970). The degree of attainment of only two of these goals 
can be estimated by the use of standardized achievement tests. 

Initial drafts of the criteria were prepared, discussed, and 
reviewed by a number of AIR staff. In addition to Impact criteria 
covering scope and effectiveness, definitional criteria were Included 
to distinguish educational products from non-products for the pur- 
poses of the study. The effectiveness or effects criteria were 
stated In terns of "accepted educational goals" so that both cogni- 
tive and affective goals would be admissible, and provisions were 
made for the effects of products whose Impact Is currently not 
directly measurable. 

The final definitional and Impact criteria, as approved by USOE, 
are given below. 
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Def tnttional Criteria 



Definitional criteria are those which were applied initially 
to distinguish educational products from non-products for the pur- 
pose of study delimitation. These criteria insured that products 
were construed as having behavioral as well as physica attr u es 
and as having evolved from empirical research and developmental 
procedures. While this is an OE supported project, proprietary 
products as well as products of OE supported projects were con- 
sidered . 



1. The vroduot should have explioit and well-defvned goods 
and obgeotives. Or^ the produot should have mpUo%t 
obgeotives of major importance. Written formulation of 
goals and/or objectives may appear as statements of pur- 
pose or outcome. Goals should relate clearly to some 
broad conceptual scheme (such as curriculum) or a 
methodology (such as a teaching-learning technique). 
Objectives should indicate the target audience, condi- 
tions, and anticipated indicators of success. 

2. The product should have procedures and guidelines for 
its implementation and use. And, the product should 
specify the texts, equipment, or techniques which are 
to be employed. The product may be a discrete unit 
or it may be a complex of things that are seen as 
integral and coherent. 

3. Systematic data regarding the development and use of 
the produot in the field should be availdble. It 
should be possible to obtain some type of existing 
systematic data regarding the product's development 
and use in the field so that the achievement of goals 
or objectives could be assessed. A high degree of 
behavioral specificity would be desirable. 



Impact Criteria 

Impact criteria are those which were applied to determine the 
Importance of the product as a contribution to educational practice. 
This importance was demonstrated along two dimensions; the scope 
or magnitude, and the effect or intensity of intended outcomes. 

These criteria insured that comprehensive products having impact on 
a large scale, over time, were included and that these products 
have demonstrable results on the target audience. 

1. Scope 

a. The produot must be implemented in one of the 
following grades: K-l-2-3-4-5-6-7-8-9-10-11-12, 

and in one of the following areas: language, 

mathematios, social studies, science, or voca- 
tional education. The product must be implemented 





at some point within the range of grades and 
subject matter areas specified above. The pro- 
duct cannot be college level, conducted in 
private or by correspondence, or preschool 
when not in association with the school program. 

Either it involves content in one of the mentioned 
disciplines or It is a process applicable to these 
disciplines. 

b. The product must have presently available a full 
and complete written description in sufficiently 
definitive form that it can be installed as 
described. A "full and complete description" 
would include Information of the type categorized 
under the heading Definitional Criteria, above. 

c. The product should be in use in at least five 
schools having no direct connection with the 
original developers. The Intent of this cri- 
teria is to exclude those products that work 
due primarily to the efforts of one person. 

d. The tajrget population at which the product is 
aimed should be a substantial one. "Substan- 
tial" was defined as at least a fourth of the 
total population In U.S. schools at the Intended 
age or grade level. 

e. The product should have been brought to the 
attention of the educational profession through 
•professional journals^ publications ^ conventions, 
or other broad dissemination procedures such as 
marketing techniques. Reference to the product 
should exist in the literature in more than one 
journal at a minimum, and its existence should 
be "known" by some reasonable proportion of 
professionals working In the discipline. 

f. The product should have come into use in the 
schools during the last five mars. A product 
could be "emerging" at the present time or could 
have been In existence at any time during the 
last five years. 

2. Effects (the product should meet at least one of the follow- 
ing two conditions) . 

a. The product should have produced results which 
suggest a measurable gain toward accepted educa- 
tional goals relative to a specified baseline and 
these results should be sustained over time. 

Measurable gain should be Indicated by the results 
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of a controlled experiment in which the norm or 
starting point for the target audience was given 
and in which comparative data over time was 
obtained. 

b. If the product impact is not directly measurable 
in terns of educational goals ^ the product should 
ha:0e contributed toward accepted educational 
goals in at least one of the following areas or 
facilitating factors: School organizational 

efficiency , classroom climate or operation^ 
learning procedures or mthodology^ and improved 
perceptual-motor skills. The "areas" listed will, 
presumably, have a bearing on student development 
by modifying environment, human interactions, or 
skills when the product is used. "Learning pro- 
cedures or methodology" included students learning 
about themselves or the world of work. 



Development of a Product Rating Form and a Product Description Form 

To obtain a broad range of products meeting the initial criteria, 
it was decided to use, as one of a number of strategies described 
below, nominating panels representing various professions in education, 
professionals whose experience would enable them to identify products 
meeting the initial criteria. To economically obtain product nomina- 
tions from panelists across the United States, mailed nominations 
were indicated. 

The need for a convenient but detailed check list to initially 
determine whether each product nominated met the criteria for this 
study led to the development of a Product Rating Form (see Appendix 
B). The form was designed so that it: would specify all of the 

definitional and impact criteria presented above; could be quickly 
filled out; and could be quickly scanned to determine whether products 
did, in fact, meet the criteria. The form was constrained to one 
page. The criteria were reduced to simple questions that could 
usually be answered by either yes , no , or don’t know . In the case 
of grade level implemented, content area, percentage of target 
population, and number of schools and students, either selected 
alternatives or space was provided so that the respondent could 
indicate the appropriate information. 

The rating form was used by both project staff and the nomina- 
tors selected from the various educational professions. When used 
by nominators, the follov.’ing materials accompanied the rating forms: 
a summary of the study's objectives and approach; and instructions 
for the rating form, which included a statement of the criteria and 
explanatory notes for each criterion. These materials were first 
pretested with AIR staff who were unfamiliar with the study, and 
resulting feedback was then used to revise them accordingly. The 
final version of these materials appears in Appendix A. 




•4 




The Product Rating Forms were filled out for all products that 
were identified during the search procedures described below and 
that appeared to be suitable products for the purposes of this study. 
Products appeared to be suitable if available infoinnation suggested 
they would meet the definitional and impact criteria and related 
cut-off points. If a product qualified under impact criterion 2b 
instead of 2a (i.e. , product impact was not directly measurable but 
it contributed toward accepted educational goals) , it was considered 
only if expected desirable student behavior was specified. As a 
result, products not oriented toward changing student behavior were 
eliminated. 

It became evident that although the Product Rating Form was use- 
ful in determining if a product met the. basic criteria for inclusion, 
additional detailed product information was needed. Thus, a Product 
Description Form (see Appendix C) , designed for staff use, was 
developed. This form, in addition to that infoimiation noted on the 
Product Rating Form, included such categories of information as: 

1. The focus of the product in terms of subject matter 
Ce.g., Language, Mathematics) or facilitating fac- 
tors (i.e., school organizational efficiency, class- 
room climate, learning procedures, or perceptual- 
motor skills) . 

2. The objectives of the product including short- and 
long-range goals. 

3. Format — whether the product consisted of textbooks, 
workbooks, television presentations, films, charts, 
games, computer programs, etc. 

4. Training requirements for personnel who implement 
the product. 

5. A summary of the development and diffusion schedule. 

6. Measured effects in terms of student outcomes with 
regard to both cognitive and affective gains. 

The Product Description Forms were used primarily as a means to 
summarize, in detail, all available information after a Product Rat- 
ing Form was filled out. 



Search Strategies Employed to Identify Products 

To identify potentially suitable products, the following search 
procedures were undertaken: .. 

1. A review of the professional literature. 
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2. A review of selected USOE reports and records not 
available through standard channels. 

3. A review of the exemplary program data collected 
by Hawkridge, et al, in the areas of vocational 
education and education for the disadvantaged. 

4. The data from the 1960 Project TALENT Survey was 
used In conjunction with the data from the 1970 
Project In Education Survey to Identify local dis- 
tricts In which gains Indicated the possible use of 
effective products. 

5. A number of projects conducted by the American 
Institutes for Research which examined the Impact 
of various educational programs were reviewed. 

6. Selected nominators, referred to above, repre- 
senting various professional groups throughout the 
United States were asked to identify products. 

A more detailed description of each of these search strategies follows. 

A Review of the Professional Literature 



As an initial step in finding leads to or identifying potentially 
suitable products, a computer search was made via the ERIC/DIALOG 
Online Retrieval System at the Region IX Office of Education in San 
Francisco. The computer contained an ERIC data base of more than 
25,000 research and research-related reports from issues of Research 
in Education through March 1970. Documents published earlier than 
1965 were not considered. The final body of documents totaled 300 
relatively recent reports and journal articles. 

In a similar fashion, an additional computer search was under- 
taken to cover two supplementary sources of potential products. One 
course, PACE (Projects to Advance Creativity in Education), included 
completed projects and the other source. Current Projects in Progress, 
included uncompleted projects. In this search, 101 entries were 
identified by the computer. 

The project was also discussed with staff from the Smithsonian 
Science Information Exchange. The Exchange prepared 150 "Notices of 
Research Projects" which described educational studies relevant to 
the project criteria. 

Project staff supplemented the above searches by a review of the 
current professional literature using the resources of the Stanford 
University and AIR libraries. A substantial number of leads to 
potentially suitable products were found through this review. Thus, 
581 leads to potentially suitable products were obtained by a review 
of the professional literature published during the last five years. 



The staff followed up these leads by contacting the developer when 
additional Information was needed to complete the Product Rating 
Form. 

A Review of Selected USOE Reports and Records 

A review of selected USOE reports, records, and product Identifi- 
cation systems, not available through standard channels, was another 
procedure employed to find leads to or to Identify potentially suitable 
products. ALERT, an Information system designed to keep school 
personnel abreast of the latest Alternatives for ^earning through 
Educational Research and Technology, under development at the Far 
West Regional Educational Laboratory, was examined. The purpose of 
ALERT Is to Increase the awareness , understanding and decision- 
oriented evaluation of new educational programs. The primary target 
audience Is the "school decision makers" — administrators, teachers, 
students, parents, and community spokesmen. Although the system Is 
not currently available to public school personnel, project staff were 
able to review approximately 75 product descriptions. Seventy of these 
provided leads to potentially suitable products. 

The Information Office for the network of R&D centers and educa- 
tional laboratories has also developed a product Identification sum- 
mary system. The center, located In Denver, Colorado, continually 
develops descriptions of products as they are being developed by the 
various centers and laboratories. Project staff were able to obtain 
from the Information Office all product summaries that had been written 
before November 1970. These summaries provided 185 leads to poten- 
tially suitable educational products. 

Several Internal documents prepared In the National Center for 
Educational Research and Development (NCERD) were also reviewed. These 
Identified various sets of products, totaling 30, selected for differ- 
ent purposes than the present project. However, these products also 
appeared to be potentially suitable products for the present study. 

Thus, a review of the ALERT system, summaries from the Informa- 
tion Office for the network of R&D centers and educational laboratories, 
and several Internal documents prepared by NCERD yielded 290 leads to 
potentially suitable products. In most cases, additional Information 
was needed to complete the Product Rating Forms for the products that 
appeared to meet the basic criteria. The staff then contacted the 
developer of each product as necessary and obtained the necessary 
Information. 

A Review of the Exemplary Program Data Collected by Hawkridge, et al 

As another strategy, exemplary programs evaluated by Hawkridge 
and his collaborators In the areas of vocational education and educa- 
tion for the disadvantaged were reviewed. It was anticipated that, 
since these programs reportedly contributed to student gains, they 
may have employed educational products suitable for consideration In 
the present study. The following program reports were carefully 
examined : 
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A Study of Selected Exemplary Programs for the Education 
of Disadvantaged Children (Hawkridge et al, 1968). The 
purpose of this study was to identify and describe success- 
ful educational programs for culturally disadvantaged 
children. Successful educational programs were defined 
as those which contributed to measured gains in cognitive 
achievement. Twenty programs were described in the 
report . 

2. A Study of Further Selected Exemplary Programs for the 
Education of Disadvantaged Children (Hawkridge et al, 

1969) . This study was a sequel to the one above, and 
the aims were identical. Eleven additional programs 
were described. 

3. A Study of Selected Programs for Vocational Education 

in Secondary Schools (Hawkridge et al, 1970) . The aims 
of this study were to identify and describe vocational 
education programs at the secondary level that had been 
successful in increasing the placement rate of treatment 
students, compared with students enrolled in other courses 
of instruction. Although a pool of 445 programs were 
identified, no program was described in sufficient detail 
to identify potentially useful products. Furthermore, 
none of the 445 programs met the original study's 
criteria. 

A review of the exemplary program data collected by Hawkridge and 
his collaborators resulted in 32 leads to potentially suitable products. 
In each instance in which the programs indicated potentially useful 
products, as defined for the purposes of this study, further informa- 
tion was obtained from the files developed for the projects. As 
necessary, additional information was then obtained by contacting the 
developer of the particular product. 

A Review of the Progress in Education Survey 

Another method for identifying educational products involved the 
search for research and development products used in high schools in 
which students had demonstrated group gains in reading comprehension. 
This was done by using data from the 1960 Project TALENT Survey con- 
ducted by the University of Pittsburgh and the American Institutes for 
Research (Flanagan et al, 1964; Flanagan, 1971), in conjunction with 
additional data derived from the 1970 Progress in Education Survey 
conducted by AIR (Jung, 1971). The 1970 survey used the TALENT reading 
comprehension test to determine whether a measurable increase in read- 
ing skill had occurred during the decade of 1960-1970. It was felt 
that this would provide one of the most reliable indicators of improve- 
ment in educational programs since 1960. 

In 1970, 12,722 students from 134 schools which had participated 
in the 1960 Project TALENT Survey were identified. The mean reading 
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comprehension raw scores for each school were calculated for 1960 
and 1970. The changes in the means from 1960 to 1970 were also 
calculated. Only those schools whose change scores equaled at least 
one-third of their standard deviations were Included for further 
examination. It was assumed, as a rough rule of thumb, that change 
scores of less magnitude would not warrant further consideration 
for the purposes of the present study. Thirty schools, or about 
25% of the total sample, met this criterion. Of the 30 schools, 
those in which the number of students who participated in both sur- 
veys totaled less than 20, and/or those in which the principals 
Indicated that the socioeconomic level of the community was higher 
in 1970 than in 1960, were excluded from further examination. Eight 
schools were thus excluded. 

The remaining 22 schools were queried for information regarding 
possible products introduced during the last five years, products 
which may have contributed to the reading achievement gains registered 
by their students. Principals of each of these schools were informed 
that their students registered gains in reading achievement from 1960 
to 1970 that appeared greater than those achieved by students in many 
of the other schools participating in the survey. The principal was 
then given a brief description of the present project and was asked 
to identify some of the more effective educational products that his 
school had introduced during the last five years. Enclosed with the 
cover letter were five Product Rating Forms noting the criteria for 
suitable products for the present study. Approximately 50% of the 
principals responded. Several indicated that they had not used dur- 
ing the last five years any new products meeting our criteria. 

Principals who did not respond were telephoned and asked to nominate 
products over the phone. Only 15 leads to potentially suitable pro- 
ducts were obtained. For each case in which a potentially suitable 
product was identified, information was then obtained from the developer 
or the disseminator of the product. 

A Review of Projects at the American Institutes for Research 

A number of projects currently being conducted or recently com- 
pleted at the American Institutes for Research examined some aspects 
of the impact of various educational programs. These projects were 
reviewed to see if suitable educational products could be identified. 
Projects reviewed were: 

1. The Assessment of Exemplary Programs for the White 
House Conference on Children (Shanner, 1970). The 
purposes of this project were to describe and analyze 
programs in terms of objectives, content, method, 
costs and benefits; and to recommend procedures to 
facilitate visits by educators to exemplary sites. 

2. Implementation of the Cooperative Longitudinal Study 
of Demonstration Education Programs (Chalupsky, 

1970). This project is aimed at identifying the 
innovative components of large-scale innovative 
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programs which appear to exhibit the greatest 
impact on students in elementary and secondary 
schools . 

3. Further Examination of Exemplary Programs for 
Ed^ucating Disadvantaged Children (Wargo, 1971). 

The objective of this study is to identify and 
disseminate information about compensatory educa” 
tion programs which have been successful in pro- 
ducing measured benefits of cognitive achievement. 

4. The Development and V alidation of an Evaluation 
System for Ascert aining the Effectiveness of ^uca- 
tlonal Lal'uratories and Research and Development 
Centers (Dunn, 1970). This project is aimed at 
developing and pilot testing a comprehensive data 
collection and reporting system, compatible with 
current Office of Education procedures. The pro- 
posed system will allow detailed analyses of the 
degree to which activities supported by the National 
Center for Educational Research and Development are 
meeting their goals. 

5. The Evaluation of Selected Teacher Training Programs 
(Tallmadge, 1971). This study was designed to identify 
various patterns of successful teacher training and to 
analyze those program characteristics contributing to 
success. 

For each of these projects, comprehensive files containing a variety 
of information had been developed. These materials were carefully 
examined to identify potentially suitable educational products and to 
obtain sufficient information to complete the Product Rating Forms. 

Sixty leads to potentially suitable products were found. 

If still further information was needed to complete the Product 
Rating Forms, the product developer and/or disseminator were contacted 
directly. 

Nominations from the Educational Professions ^ 

Independently of the above procedures, products were also identified 
by selected nominators representing various educational professional 
groups throughout the United States. A list of 200 potential nominators 
from a variety of professional interests and institutions was identified. 
Included were representatives of the regional educational laboratories, 
research and development centers, policy research centers, regional 
offices of the United States Office of Education, centers for voca- 
tional education, major universities, state departments of education, 
selected school districts., non-profit private organizations, private 
foundations, profit organizations, major educational publishers, and 
representatives of the Department of Health, Education, and Welfare. 



One hundred nominators were initially selected from the pool of 
200 to reconunend products. Government, profit-making, and private 
non-profit agencies were equally represented in the selected group 
of 100 nominators. These nominators were contacted initially by 
telephone to obtain an advance commitment to identify products. In 
some cases, the nominator directed the project staff to another mem- 
ber of his organization (in these cases, this member was then tele- 
phoned). During the telephone conversations, the staff briefly 
described the scope and purpose of this study and outlined the 
services requested of the panelist. Once the advance commitment 
was obtained, the panelist was sent a cover letter, instructions 
for rating forms, and 10 Product Rating Forms. The cover letter 
briefly summarized the project’s objectives and purposes and the 
task required of the nominator. The Instructions for the rating 
forms included detailed descriptions of the criteria for selecting 
educational products. Samples of the cover letter and instructions 
are contained in Appendix A. Each of the 100 nominators were con- 
tacted by both telephone and letter within a two-week period. 

If the rating forms were not returned within a two-week period, 
follow-up telephone calls were made. During these follow-up calls, 
the staff queried the nominator as to whether the forms had been 
received and whether he had had a chance to review and fill out the 
forms sent. Again, in some cases, the nominator had passed the forms 
on to another member of his organization. In such cases , the staff 
contacted the new representatives and urged them to fill out and 
return the Product Rating Forms. If the rating forms had not been 
returned within a second two-week period, the nominator was again 
telephoned. At this time he was asked to nominate products over the 
telephone and a staff member filled out the Product Rating Forms 
during the telephone interview. 

Eighty percent (80%) of the nominators responded within six weeks 
from the date of the initial contact. Some of the difficulties usually 
associated with mailed questionnaires were attenuated by the initial 
personal telephone communication prior to any mailing. The follow-up 
calls were equally useful. If the Product Rating Forms were not com- 
pletely filled out, or if the nominator circled an "I don't know," 
the original developer or the disseminator was then contacted to 
obtain the additional needed information. While many Product Rating 
Forms were returned, only 60 unique products were identified through 
this search procedure, as nominators frequently identified the same 
products . 



Consultant Conference 



A conference was held December 14-15* 1970, at the Palo Alto 
office of the American Institutes for Research. The overall purpose 
of the conference was to review this study. The objectives of the 
conference were to review the selection criteria, the products 
identified to date, and proposed case study procedures, and make 
recommendations for further activities. 
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Participants at the conference ropresented government, proflt- 
raaklng, and private non-profit agencies and were: Egon G. Cuba, 

Associate Dean, School of Education, University of Indiana; Arthur 
A. Lumsdalne, Professor of Psychology, University of Washington; 

Norman Boyan, Dean of the School of Education, University of 
California at Santa Barbara; Robert Mager, Consultant to the 
American Institutes for Research; William V. Clemans, Vice President, 
Educational Systems, Science Research Associates; David Engler , 
Instructional Technology, McGraw-Hill Book Company; and John C. 

Flanagan, President, American Institutes for Research. In addition, 
James Becker, Director of Research for Better Schools, Philadelphia, 
Pennsylvania, participated In a similar review and discussion with 
the staff, but was unable to attend during the conference dates. 

During the two-day conference, one day was devoted to the subject 
of product selection and the other day to a discussion of case study 
procedures. Several suggestions were made pointing to further clari- 
fication of the criteria for product selection and more precise defini- 
tions of the major concepts Included In the criteria. In most cases, 
consultant suggestions resulted In additional explanatory notes supple- 
menting the original criteria. For those criteria for which one of 
several cut-off points could be selected, the consultants tended to 
offer differing recommendations. In general, the consultants from 
profit-making sources suggested large-scale usage by schools and num- 
ber of students, while university-based and research and development 
oriented consultants suggested that product effectiveness In tex' s of 
demonstrated student gain was more critical than widespread adoption. 

It was generally agreed that products with an available research and 
development history and with reasonably "hard" data on student gain In 
either cognitive or affective areas should be given weight for Inclu- 
sion, other factors being equal. 

Considerable discussion was generated regarding products which are 
not directly related to student classroom behavior, such as products 
focusing on teacher training or modular scheduling. A number of sugges- 
tions were made as to both areas that might yield useful products and 
particular products that were not Included In the Initial list at that 
time. Such suggestions were followed by the staff and yielded 30 leads 
to potentially suitable products. 

The consultants also reviewed the tentative classes of variables 
scheduled to be Identified In the case studies. They warned that a 
tight and structured Interview schedule could color the results, and 
recommended that Information gathering be as open-ended as possible In 
a form that would allow what actually happened to emerge. Developers 
who feel they were successful may forget all the problems they had, 
while those who were unsuccessful may give Invalid reasons for their 
failures. Although problems related to management or personality 
conflicts wlfhln an organization may be Important factors In the his- 
tory of a product, this Information may be difficult to obtain. Every 
effort should also- be made to Identify organizations started on an 
opportunistic basis. 
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It was noted by the consultants that complete reliance on the 
developer and disseminator of a product for information might be 
questionable since they have an abvlous bias and may not report data 
which reflect unfavorably on their product. It was suggested that an 
attempt be made to obtain data on the effectiveness of the products 
from sources other than the developer. If data from one source contra- 
dicts other information on the product, it is necessary to study care- 
fully how the evaluations were designed and conducted in order to 
decide how much confidence to have in the results. 

Regarding the availability of Information, the consultants 
pointed out that data on product effectiveness gathered by someone 
other than the developer will be the hardest to obtain. In addition, 
although some developers may have Information on the development of 
their products, it may not have been analyzed or reduced to a useful 
format. It was felt that few commercial publishers may have the kind 
of developmental information being sought. Publishers rarely conduct 
any systematic evaluations of products being developed, and data 
informally gathered are not reduced to a readable form. 

All consultants agreed that the trend exemplified in the pre- 
sent study of examining in detail the research, development, and 
diffusion history of educational products represented a healthy trend 
in education. Some doubt was expressed as to how many general conclu- 
sions could be derived, considering the small number of cases and the 
large number of variables involved both within and between products. 

The consultants also concluded that since other studies have dealt 
with adoption and the effectiveness of a product, this study can make 
the most valuable contribution by concentrating on the developmental 
history of products. 



Selection of Products for Case Studies 





Initial Selection of Exemplary Products 

As discussed in greater detail in the Results section below, the 
application of the definitional and Impact criteria presented above to 
the list of products that had been Identified resulted in the location 
of 117 products that met the criteria. Criteria for the selection of 
about 20 exemplary products on which case studies would be made were 
then developed. It was initially anticipated that the specific criteria 
for the selection of the exemplar products would Include the following: 

1. About two -thirds of the products would be "tangible," 

(i.e., a specific thing such as a curriculum, work- 
book series, TV course, etc.), the other third "less 
tangible" (including techniques, organizational con- 
cepts, and procedures such as team teaching, systems 
for Individualizing instruction, etc.). 

2. The products selected would represent the various 
source categories from which such products are made 
available to the schools . 
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3. Products representing various target populations 
would be selected to improve chances of securing 
generalized findings. 

4. A few products for which the initial educational 
impact seems to have disappeared would be included. 

Detailed consideration of these potential criteria uncovered a number 
of problems, such as difficulty working within the "tangible, less 
tangible" distinction. An alternate procedure was, therefore, 
employed . 

As a first step in identifying a pool of exemplars from the 117 
products meeting the definitional and impact criteria, two further 
criteria were imposed. These were extent of use and evidence of 
student gain. To quantify extent of use, a three-point scale was 
used to identify high, moderate, and low use. The products used in 
5-24 schools were rated low, products used in 25-99 schools were 
rated moderate, and products used in 100 schools were rated high. 

For evidence of student gain, products for which one type of gain 
(e.g., cognitive student gain, affective student gain, or contribu- 
tions through facilitating factors) was indicated by only a 
nominator were rated low. Products for which more than one type of 
gain was indicated by a nominator were rated moderate. Products for 
which gain was indicated by a nominator and for which there was sup- 
porting evidence suggesting a carefully controlled study were given 
a high rating on gain. If a product was rated as moderate or high 
on one of these two criteria, it was retained in the pool. 

The use of these two additional criteria yielded a fairly large 
pool of potential products. The following constraints were, there- 
fore, imposed to reduce the available pool: 

1. Focus of the product. The two overall kinds of focus 
in these products were either subject matter or fac- 
tors facilitating learning, but not directly related 
to a specific subject matter. This latter focus 
included such purposes as organizational efficiency, 
classroom climate or operation, learning procedures 
or methodology, and improved perceptual— motor skills. 

An attempt was made to get approximately equal repre- 
sentation of products directed in each of the five 
content areas of mathematics, science, social studies, 
language, and vocational education. In addition, a 
representation of products dealing with the facili- 
tating conditions which would be approximately equal 
to the number of products in each of the major con- 
tnet areas was sought. Thus, a successful applica- 
tion of focus balancing would result in perhaps four 
products in language, four in mathematics, four in 
vocational education, four dealing with facilitating 
conditions, etc. This constraint was applied success- 
fully except for vocational education products. There 



were simply not a sufficient number of such products 
that met the basic definitional and impact criteria. 

2. Source of development. An attempt was made to repre- 
sent the three major sources of product development; 
i.e., government agencies, private non-profit firms, 
and profit firms. Although an attempt was made to 
represent these three groups equally, this was not 
feasible in light of the original pool of 117 pro- 
ducts. Representation of government sponsored devel- 
opers about equaled the combined representation of 
private non-profit and profit firms. 

3. Format. An attempt was made to select the final set 
of products so that a variety of formats would be 
represented. These include the range from a simple 
textbook and manual to a profuse diversity of audio- 
visual, game, and other relatively exotic materials. 

4. Grade level. An attempt was made to obtain representa- 
tion of products directed at every range of grade level 
from K-12. Thus, some products were included that 
covered the entire K-12 range; others included only K 
through the primary grades; others the intermediate 
grades; others high school. An attempt was made to 
balance this so that the bulk of the products were 
neither in the primary grades nor in high school; nor 
were the K-12 all-inclusive types of products. 

5. Visibility. In those cases in which two or more pro- 
ducts met the above criteria and constraints and it was 
necessary to select one product, weight was given to 
the product with the most "visibility." These were 
products that had attracted a great deal of attention 
within and sometimes without the educational profession. 

Often they were products that had recently been, or are 
currently being hailed as emblematic of a new movement 
or programmatic effort. 

As noted above, it was considered desirable to include in the pro- 
ducts selected for case study a few for which the initial educational 
Impact seems to have disappeared. It was, however, impossible to 
locate such products with certainty since attenuated impact is diffi- 
cult to define. In addition, all products nominated had met the impact 
criterion, and the initial information available on each product did 
not permit the identification of lost impact. Often times products may 
appear to have suffered a temporary loss of impact which is then re- 
gained with a revised form of the product. An attempt was made, then, 
to locate for this category products that are growing very slowly in 
comparison tp others, or products for which no further developmental 
or revision funds appeared to be available'. 
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Selection of 21 Products From Among the Exemplars 



Application of the above procedure resulted in the identifi- 
cation of 33 exemplary products. The following criteria were 
then applied to select 20 exemplary products for case study, and 
13 alternates: 

1. Products were grouped according to the focus. Pro- 
ducts within a focus group that had the lowest rat- 
ing on extent of use and student gain were placed 
on the alternate list. 

2. An attempt was made to maintain the balance across 
grade levels and content areas. 

3. If other factors were approximately equal, visi- 
bility was given additional weight. 

4. Two products which appeared to meet the criteria 
for possibly attenuated impact were included, plus 
one product that appeared to have lost impact 
initially but then regained it in a revised form. 

Project PLAN, developed by the American Institutes for Research 
and the Westinghouse Learning Corporation, had met all of the selec- 
tion criteria but had been excluded from consideration since this study 
was being conducted by AIR. After 20 exemplary products were selected 
for case study in fulfillment of the contractual obligations with USOE, 
PLAN was added as the 21st product. As noted below, this gave the 
opportunity to develop and refine case study procedures before they 
were applied to the 20 other products. 

Rationale for Changes in the List of 21 Exemplars 

As the project proceeded, four products on the initial list of 21 
had to be dropped and replaced with products from the alternates. One 
product was dropped because no cooperation could be obtained from the 
developer /disseminator. After several telephone calls, it was obvious 
that much of the information needed would not be retrievable. The 
decision to drop the product was made before a letter, indicating the 
selection of it as one of the 21 exemplars, was sent. For a second 
product, the originator and developer did not want to participate. He 
had severed ties with the R&D center where he developed~^the product, 
but the product remained in the hands of the R&D center.' This critical 
staff member wished not to support or be involved in any R&D center 
related efforts. Consequently, the product was dropped from the list 
as there was no staff member at the R&D center who could adequately 
provide the needed information. This decision was made after a letter, 
indicating selection of the product as one of the exemplars, was sent 
to the developer. 

Absolutely no cooperation could be obtained from the developer of 
a third product. Many attempts, including a series of telephone calls. 
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were made to obtain a commitment to participate. This occurred 
after a letter, indicating selection of the product as one of the 
exemplars, was sent. The developer was never "available." Conse- 
quently, the product was dropped from the list. A fourth product 
on the original list was dropped because one of the selected 
alternate products was from the same organization that had already 
had one product selected. 

The reasons for selecting specific alternate products to replace 
the four that were dropped were noted. One product, which was highly 
rated on several of the most critical criteria, included activities 
similar to a product that was dropped. A second product, which was 
also highly rated especially on the scope criteria, provided a third 
social studies curriculum to parallel the three in science and 
language. A third provided a solid language product that was rated 
very highly on scope criteria, while a fourth provided a good replace- 
ment for a dropped mathematics product and in addition represented one 
of the few available programs of its type. 

A list of the finally selected 21 products is given on page 37 
in the Results section, below. 



Development of Procedures for Collecting Case Study Information 

The procedures for collecting case study information were developed 
Iteratively in conjunction with determining the components of the case 
studies and actually trying out the techniques on the first few pro- 
ducts. This section outlines the Intended attributes for the case 
studies and the components that would be Included in each case study. 

A report on the evaluation of the components of the case studies is 
then presented, followed by the major steps employed in collecting 
case study Information. 

Intended Attributes of Case Studies 



It was Intended that each systematic case study of the 21 exemplary 
products would possess the following attributes: 

1. It would present the evolution of a product in fine- 
grained detail. Such an account would present a more 
in-depth view than is ordinarily available in the 
literature. 

2. It would contain an accurate reconstruction of the 
ac tual steps followed in the development and diffu- 
sion of the product. 

3. It would Include those variables of interest in 
current thinking about Instructional development 
and diffusion. 



4. It would encompass a critical summary of all avail- 
able evaluations of the product conducted before the 
product was made generally available to users, i.e., 
evaluations during development, including field test- 
ing. 

Each of these attributes was to be obtained through the case study 
procedures which were to be followed in tracing the history of each 
educational product from its origin to its adoption and utilization. 
The specific procedures for obtaining information for the case 
studies were developed accordingly. 

Components of the Case Studies 

It was initially anticipated that the systematic case studies 
were to Include the following information as a minimum: 

1. Origin of the product — where the idea came from. 

2. The design and conceptualization of the innovation. 

3. History o:.' the development of the product, including 
the source of support and the identity of the 
developers . 

4. Information on field or pilot testings of the product, 
and the results of such analyses. 

5. Efforts in the dissemination of information about the 
product . 

6. Information and data that might serve as direct or 
indirect measures of the impact of the product in its 
operational form; included would be data on the num- 
ber of students involved, the nximber of items sold, 
any surveys of student and teacher reactions , 
summaries of evaluative studies aimed at investiga- 
tions, etc. 

In considering the best approach to recording and presenting the 
information that was to be collected, it was decided that the developed 
case study for each product would include four components: 

1. A systematic narrative history. 

2. A product data record. This was to consist of a 

systematic coded matrix of those quantifiable aspects 
of products: quantifiable aspects of design, develop- 

ment, diffusion and adoption, including census-type 
information, product characteristics, and evaluative 
data. Information in such a matrix was to be arranged 
and coded for possible key-punching and machine treat- 
ment of the data. 
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3. An analysis of the developmental sequence of each 
product. This was to be summarized In a flow-diagram 
analysis of the sequence of major events In each 
product's developmental history. 

4. A description of the critical decisions In the history 
of the product. 

How the actual design of each of these components evolved Is described 
below . 

Evolution of the Components of the Case Studies 

Initiation of a Master Outline for narrative history . Based upon 
the kinds of Information that It was anticipated the narrative history 
of each product would Include, a Master Outline was Initiated. It 
was assumed that the Master Outline would not only guide the site 
Interviewers, but would also be used In preparing the report on each 
product. Initially, the critical decision Information, the major 
event flow diagram, and the product data record were planned as 
separate documents not to be Integrated Into the narrative history. 
Information in the narrative history was to focus on the origins, the 
development, the evaluation, the diffusion, and the adoption of the 
product, and a brief produ^.t description. 

As more and more Information was specified as necessary for com- 
pleting a narrative history of a product, it became apparent that more 
guidelines were necessary than initially envisioned. Guidelines were 
developed to explain what each section of the narrative history should 
Include, and how such information could best be obtained. The develop- 
ment of explanatory notes and the suggestion of interview strategies 
led to an amplification of the Master Outline. The additional explora- 
tion and questioning resulted In more and more detail In the outline. 

Trial of techniques during training of site Interviewers . With an 
Intermediate version of the outline for the narrative history and an 
understanding that quantifiable data would be needed to complete the 
Product Data Record, that some fcrm of major event flow diagram would 
be desirable, and that a documentation of the critical decisions would 
be informative, staff members pilot tested the strategies to be 
employed in site interviews. First, in role playing situations, staff 
members, playing the part of the product developer or the interviewer, 
tested the procedures designed to obtain the necessary information. 

This step led to modifications in interview techniques and to an inter- 
mediate version of the Master Outline. A second and final exercise 
involved the application of the search procedures to Project PLAN, 
which had been developed at AIR. All potential site team members 
reviewed documentation on PLAN, and conducted interviews with key 
personnel. Each key person interviewed presented the staff with a 
critique of his approach and technique. Again, modifications were 
made in both the search procedures and the Master Outline. 
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Modification of ease study components . As the site visits were 
being conducted, more modifications were being made In the design of 
the narrative history, the major event flow chart or diagram, the 
critical decision description, and the Product Data Record. 

The major event flow diagram was to provide the reader of the 
report with a convenient symbolic overview of the major activities 
in the product's history. To achieve this objective, the flow chart 
was to consist of two parts: a symbolic illustration and a narrative 
describing the illustration. The objective of the flow chart was not 
changed. However, the flow chart was made self-explanatory so that a 
narrative, other than that found in the text of the narrative history, 
was not needed. 

The critical decision description had a similar history. The 
purpose of the critical decision description was to provide the reader 
with a convenient overview of the critical decisions made in the his- 
tory of the product. It was to consist of two parts: a symbolic 

diagram and a narrative describing the diagram. The objectives never 
changed, but the symbolic diagrams were never developed as a straight- 
forward narrative was found to be the most effective means of present- 
ing the overview of the critical turning points. The narratives came 
to include: the decisions that had to be made, the alternatives avail- 

able for each decision, the alternative selected, the forces leading to 
the selection of a particular alternative, and the consequences result- 
ing from the selected alternative. 

In considering the most efficient method for reporting information 
on the case studies with the simplifications noted above, it was decided 
to include the narrative history, flow chart, *.nd critical decisions 
narrative for each product in one document called a Product Development 
Report. It was further decided that the Product Development Reports 
would be prepared as separate documents rather than appendices to the 
final r€;port. A list of the Product Development Reports is given in 
Appendix H. In addition, the Master Outline for the narrative history 
was reworked to become the Master Outline for the Product Development 
Reports. The Outline, which appears in Appendix F, came to include 
sections for: product description; origin of the product; development 

of the product (the major event flow chart appears in this section or 
the section on the origin of the product); evaluation of the product; 
diffusion of the product; adoption of the product; future of the pro- 
duct; and a description of the critical decisions in the history of the 
product. 

The development of the Product Data Record, containing quantifi- 
able aspects of product design, development, diffusion, and adoption, 
paralleled the development of the Master Outline. In fact, as the 
major sections of the Master Outline were expanded, specific data ele- 
ments were identified and included on the Product Data Record. Each 
item on the Master Outline that could be reasonably quantified (i.e., 
in terms of numbers, yes or no, or a rating system) was included. When 
necessary, guidelines for filling out the Record were specified. The 
Product Data Record remained a separate component and was not an inte- 
gral part of the Product Development Report. 
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Major Steps in Collecting Case Study Information 

Initial agreement with product development and/or diffusion agency . 
Personal communication was utilized to seek cooperation and commitment 
to the case study for each exemplary product. The product agency was 
presented with an orientation to the purpose and general approach of the 
study, the information required, and the probable outcomes of the pro- 
ject. 



Retrieval of available product records . An attempt was made to 
obtain in advance, from the development and/or diffusion agency, avail- 
able records relevant to the case study of the product. The extent of 
these records varied widely from product to product. Minimally, a 
general report and some brochure-type information was obtained from the 
developer. In some cases, records Included original proposals, progress 
reports, in-house records, budget and schedule data, etc. Records from 
the diffusion agency Included sales and distribution Information; 
extent of distribution, identification of users, and date of delivery. 

Transposition of information from product records . How much infor- 
mation was retrieved and how much time was available for pre-site visit 
writing determined the extent to which the narrative history, the flow 
diagram, the critical decision description, and the Product Data Record 
were initiated. In some cases, a good draft was prepared for each of 
these components before the site visit was conducted. In most cases, 
however, these components were only in very rough form before the inter- 
view with the developer. An attempt was made to determine which informa- 
tion had been retrieved, which could be obtained from documents obtained 
during the site visit, and which could be obtained only from the devel- 
oper In the interview. Thus, all the information that could be retrieved 
on each product was utilized to: frame an initial outline of the pro- 
duct case history; sensitize site visit personnel; identify areas for 
further exploration during the interview; and provide a basis for com- 
parison with Interview information. 

Site Interviews . Arranging for economic site visitation trips 
required extensive scheduling within a product development organization, 
among the various key staff, and between different product development 
groups. Typically, the development staff on a particular product were 
involved in other projects and were frequently found to be working for 
other organizations in other locations at the time of the site interview. 
There were also cases in which the key developer had left the original 
organization and was working in an entirely different region. During 
the interviews an attempt was made to obtain information regarding 
those factors not obtainable from the written materials. When pre-site 
visit writing was very minimal, the topics discussed with the developers 
covered the entire range of factors for all the components (e.g. , the 
narrative history or the critical decision description) . All inter- 
views were tape-recorded. 

Transposition of site interview information . Upon returning to 
AIR from the site visits, the tape-recorded interviews were transcribed. 



Usually, the developers discussed factors related to each of the com- 
ponents (e.g., the major event flow diagram or the Product ilating 
Record)* the information was organized under the various components 
after the tapes were transcribed. 

Throughout the case studies, a continuous attempt was made to 
obtain the most valid retrospective information concerning the pro- 
duct’s history. To this end, a check and balance system was maintained 
and involved comparisons between product records and interview data; 
comparisons between information obtained from two or more respondents 
interviewed independently; and comparisons of information derived from 
development, diffusion, or user agencies. The development and the 
diffusion agencies were separate entitles in some cases. In other 
cases, such as the regional laboratories, these functions were served 
by one agency, or even the same group within that agency. 



Preparation of Product Development Reports and Product Data Records 
Procedures for Preparing Product Development Reports 

After the site ■ interview information was transcribed, the writer 
of a Product Development Report organized all available information by 
grouping it under the relevant sections of the report (e.g., under 
origins or under critical decisions). Once the information was grouped, 
the writer wrote each section, one by one, until the entire report was 
completed. In order to maintain uniformity from one Product Development 
Report to the next, each report was prepared following the Master Out- 
line as a guide, and using headings from it. Depending upon the 
specific product, at times lower level entries on the outline were not 
needed, and the materials were grouped under higher level headings. 
Certain sections of the outline, such as Development of Performance 
Measures/Assessment Techniques, were rarely used since the Information 
was not available above and beyond that Included in the section on 
Funding for Product Development. 

During the preparation of the Product Development Reports, in 
many cases some additional information was needed and the developer 
or disseminator of the product was contacted. In some cases the original 
draft of the report, completed before the site visit, was a good start- 
ing point. Quite often the information in the original draft was simply 
grouped under the relevant sections of the outline, just as was all 
other information. 

After a Product Development Report was completed and reviewed by 
the staff, it was sent to the developer for review. The developers 
either telephoned the revisions they thought necessary or mailed them 
in a letter to AIR. The extensiveness of the requested revisions 
varied from a few editorial comments to extensive revisions of several 
sections. In most cases the revisions requested by the developers 
were made; however, in a few Instances the requested revisions would 
have lowered the descriptiveness of the report and were not made. 
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Complete responsibility for Interpretations concerning any facet of 
development, evaluation, and diffusion rested with the authors of the 
report. 

Procedures fo r Coding Product Data Records 

The Product Data Record for each product was typically completed 
right after the site visit; It was then reviewed and modified after 
the corresponding Product Development Report was completed. Changes 
were usually made In how the elements were coded after the Product 
Development Report was completed. That Is, Information needed to 
write the Product Development Report was also needed to code the data 
elements for the particular product; this Information was not con- 
veniently available until It was Included In the various sections of 
the report. All Product Data Records were carefully reviewed to make 
certain that each element was coded as accurately as possible, given 
the available Information. The elements of the final Product Data 
Record are given In Appendix G, which was used to tabulate data for 
the 21 exemplary products for the Results section below. 



Coordination With Other Projects 

At least certain superficial similarities appeared to exist 
between the project discussed In this report and three other USOE- 
funded projects, and a continuous effort was made to communicate and 
coordinate activities among these projects. The other projects 
Included: "Generation of Information to Support Long-Term Manpower 

Studies of and Planning for Training Programs In Educational Research, 
Development, Diffusion and Evaluation" conducted by Teaching Research 
Division of the Oregon State System of Higher Education; "The Develop- 
ment and Validation of an Evaluation System for Ascertaining the Effec- 
tiveness of Educational Laboratories and Research and Development 
Centers" being conducted at the American Institutes for Research (Dunn, 
1970) ; and the Product Evaluation Project conducted at Educational 
Testing Service for the National Center for Educational Communications 
(Epstein et al, 1971). Either the directors of or the appropriate 
USOE personnel concerned with each of these other projects was con- 
tacted with regard to the objectives and procedures of the projects 
as they related to the project discussed In this report. Continuing 
communication was made so that minimal overlap and maximum productive 
complementariness of effort was maintained. 



RESULTS 



Through the search procedures described in the Method section, 
1,068 leads to potentially suitable products were found. About three- 
fourths of these products were excluded without using Product Rating 
Forms; the available information obtained indicated that these excluded 
products obviously did not meet the definitional and Impact criteria 
discussed above. Product Rating Forms were employed for 235 products; 
at that time the available information on these products indicated that 
they might meet the criteria imposed by this study. However, employ- 
ment of the criteria and related cut-off points discussed above reduced 
this number to 117. It may be noted that employing the criteria, in- 
cluding the one which indicated that products must have produced results 
which suggest a measurable gain toward accepted educational goals (or 
which suggest a contribution toward such goals), and the relevant cut- 
off points for the various criteria, did not make the final pool of 
selected products too small for the purposes of this study. 



Relative Effectiveness of Search Procedures 



The most effective search procedure employed was the review of 
selected USOE reports, records, and product identification systems. 
This included an examination of the ALERT system, product summaries 
from the Information Office for the network of R&D centers and educa- 
tional laboratories, and internal documents prepared by NCERD. The 
procedure yielded more than 100 leads, most of which were of high 
quality. That is, they frequently led to suitable products for which 
Product Rating Forms were completed. Two procedures, obtaining nomina- 
tions from the educational professions and reviewing current projects 
at the American Institutes for Research, resulted in a moderate number 
(i.e., 50-100) of leads, and most of these were of high quality. 

During the consultant meeting, only a few leads (i.e., about 29) were 
obtained, but these were mostly high quality leads. The review of the 
professional literature yielded many leads to products, but most of 
these leads were of low quality in that they frequently did not lead 
to suitable products for which a Product Rating Form was completed. 
Finally, two search procedures, reviewing the exemplary program data 
collected by Hawkridge, et al, and a review of the Progress in Educa- 
tion Survey, resulted in a few leads to products, and most of these 
were of low quality. Thus, it seems that the best procedures for 
identifying educational products with impact involve an examination 
of various information sources or product data files like the ALERT 
system and a request for nominations from members of the educational 
professions, perhaps members who were specialized in one subject matter 
area and who had summarized R&D in that area. 
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Reasons for Excluding Products for Which Product 
Rating Forms Were Employed 



Products were excluded, not for lack of merit, but for not meeting 
the specific criteria for this study. About 60% of the products ex- 
cluded were developed by government sponsored agencies, and the remain- 
ing 40% were about equally divided between private non-profit and 
profit-making organizations. Thus, the representation of the three 
types of developers in the excluded group of products was very similar 
to that in the group of selected products. Likewise, fewer products 
in the vocational education content area than in any of the other con- 
tent areas were excluded, just as the vocational education content 
area was not represented as frequently as the other content areas in 
the group of 117 selected products (see Table 3 for the percentage of 
the 117 selected products by content). 

Products for which Product Rating Forms were employed were ex- 
cluded primarily for six major reasons. About 4% of the products were 
excluded because they were implemented in pre-kindergarten classes not 
associated with a general K-12 school program. About 11% were excluded 
because they were used in less than five schools. About 5% were ex- 
cluded because the target population at which the product was aimed was 
less than a fourth of the total population in United States schools at 
tjjg intended age or grade level . Nearly 29% were excluded because they 
were still being developed and had not yet come into use in the schools. 
Finally, almost 38% were excluded because their impact was measurable 
only in terms of contributions toward accepted educational goals and 
the expected, desirable student behaviors resulting from these contri- 
butions were not specified. 



Description of the 117 Selected Products 

Each of the 117 products that passed, the criteria imposed by this 
study are described in brief paragraphs in Appendix D to give the 
reader a genera.l idea of the types of products that did pass the 
criteria. These brief paragraph descriptions typically include the 
following kinds of information; the focus of the product (subject 
matter and/or one or more of the facilitating factors— organizational 
efficiency, classroom climate or operation, learning procedures or 
methodology, and perceptual-motor skills); the content language, 
mathematics, science, social studies, and vocational education); 
grade levels covered; target population percentage; objectives; 
approach to subject area (e.g., non-graded, individualized, tradi- 
tional, etc.); and format (e.g., textbooks, workbooks, films, etc.). 
Appendix E gives a detailed description of each of the 117 selected 
products using 19 descriptions. 



Characteristics of the 117 Selected Products 



The characteristics of the 117 selected products In terms of the 
descriptors used In Appendix E are presented below. 

Focus 

Focus of a product refers to what It deals with most directly, 
subject matter content (e.g., language), and/or one or more facili- 
tating factors. As may be recalled. It Is through these facilitating 
factors that a product might contribute toward accepted educational 
goals,. A product might focus on only subject matter, only one facili- 
tating factor, or several facilitating factors, or any combination of 
subject matter and facilitating factors. 

Table 1 shows the frequencies and percents of the 117 selected 
products with the various foci or combinations of foci. Subject 
matter was by far the dominant focus. One hundred products (85%) had 
a focus of subject matter only, while 103 products (88%) included 
subject matter as a focus. Ninety— one percent of the products had a 
single rather than a multiple focus. Organizational efficiency was a 
focus of 3% of the products, classroom climate or operation and per- 
ceptual-motor skills 6%, and learning procedures or methodology 10%. 
There were no products with only organizational efficiency or class- 
room climate as a focus. 

The focus of the 117 selected products by grade level is giyen 
In Table 2. For each focus the 117 products were distributed fairly 
similarly across the 13 grade levels, although there was some tendency 
for classroom climate, learning procedures, and perceptual-motor skill 
to be more highly represented in the primary grades. 

Content 

Content of a product refers to which of the five major subject 
matter content areas (e.g., language) for which it was designed. If 
a product is focused on subject matter, the particular subject matter 
would indicate the content of the product. If a product is focused on 
a facilitating factor; e.g., learning procedures, the particular sub- 
ject matter for which the learning procedures were designed would indi 
cate the content of the product. A product may be designed for one or 
more of the five major content areas. 

Table 3 shows the subject matter content of the 117 selected 
products. Mathematics and vocational education (26% and 9% of the 
products, respectively) were not as well represented as language 
(35%), science .(39%), and social studies (42%). The relatively small 
number of nominated vocational education products had been noted early 
in the project , but adding additional nominators in this area did not 
result in a large number of such products being located and passing 
the selection criteria. About 79% of the products were designed for 
a single rather than multiple content, with a quarter of the products 



focusing on social studies only. Three percent of the products had 
subject matter content in all five areas, 9% in language, mathematics, 
science, and social studies. 

As noted in Table 4, which presents the frequencies and percents 
of the 117 selected products by grade level and content, more products 
were designed for language and mathematics at the elementary level than 
at the secondary level; similar numbers of products were designed for 
science and social studies at the two levels, and more products were 
designed for vocational education at the secondary level than at the 
elementary level. 

Format 



The format of a product refers to the mode in which information is 
presented. Formats considered weret paper products (Including text” 
books, workbooks, manuals or guides, and tests); audio-visual (tele- 
vision, films, and audio equipment); kits and equipment (including dis- 
plays, games, and laboratory equipment); and computer. A product may 
have one or more of the various types of format. Table 5 shows the 
frequencies and percents of the 117 selected products with the various 
formats. Seventy percent of the products had a multiple format rather 
than a single format. Most products (85%) included some type of paper 
product. About 40% included the audio-visual format, and a similar 
percentage Included the kit and equipment format. Only 9% of the pro- 
ducts involved the use of the computer, as may have been anticipated. 
The most frequent combination of formats (58%) was manuals or guides 
and other format (s), with textbooks and other format (s) not far behind 
(56%) . Textbooks only were used as the format for only 9% of the pro- 
ducts, while no products used only workbooks, tests, television, films, 
kits, displays, or laboratory equipment, as may have been expected. 

Grade Level 

Grade level refers to the grades (i.e. , K-12) in which the product 
was being used. Table 6 shows the frequencies and percents of the 117 
selected products designed for each grade level. Table 7 shows the 
frequencies and percents of these products by the number of grades 
covered. The various grade levels were nearly equally represented; 
however, grades one and two tended to be represented more frequently 
(51%) than the other grade levels. Very few products (19%) were 
designed for a single grade level. More products (30%) were designed 
to cover four through six grade levels than any other range of grades. 
However, a significant number of products were designed for nearly all 
grade levels . 

Percent of Target Population 

Percentage of target population refers to the percentage of the 
total population in United States schools at the intended age or grade 
level for which the product was aimed or designed. Table 8 shows the 
frequencies and percents of the 117 selected products by Intended per- 
cent of target population. The majority of the products (68%) were 
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targeted for the total population at the Intended sgs or grade level. 

Very few products (8%) had a 25% target population, which was the cut- 
off point for this variable on the impact criteria used to select 
products. 

Number of Schools and Students Using the Product 

Table 9 shows the frequencies and percents of the 117 selected 
products by the number of schools using the product. Table 10 shows the 
frequencies and percents of the 117 selected products by the number of 
students using the product. About one— half of the products were being 
used in more than 100 schools and by more than 10,000 students. A num- 
ber of products were used in more than 5,000 schools (4%) and by more 
than 100,000 students (16%). Product use in at least five schools was 
one of the criteria used in selecting products with impact. The intent 
of this criteria was to exclude those products that work due primarily 
to the efforts of one person. 

Degree of Dissemination 

Degree of dissemination refers to how frequently the product was 
brought to the attention of the educational profession through print 
channels, conventions, or other broad dissemination procedures such as 
marketing techniques. Twenty-six percent of the products were brought 
to the attention of the educational profession through more than 10 
known sources, 28% through six to nine sources, and 46% through two to 
five sources. It may be recalled that one of the criteria for selecting 
products was that reference to the product should exist in the litera- 
ture in more than one journal as a minimum, and its existence should be 
"known" by some reasonable proportion of professionals working in the 
discipline. 

Year Product Came Into Use 

Year came into use refers to the year in which the product was 
available for adoption in a school system. Table 11 shows the fre- 
quencies and percents of the 117 selected products by the year the pro- 
duct came into use. More products came into use in the later years of 
1965-70 than in the earlier years of this period, with 25% coming into 
use in 1970 compared to only 6% in 1965. It may be recalled that the 
criteria for selecting products included that the product should have 
come into use in the schools during the last five years. Only 18 pro- 
ducts came into use prior to 1967 so that possibly this criterion could 
have specified only the last three years. These data should be 
interpreted cautiously, however, since at times it was difficult to 
pinpoint the exact date a product came into use. 

Measured Gains 

EA^h individual who nominated a product was asked to indicate 
whether 'pr not the product produced results which suggested measurable 
student ^ain toward accepted educational goals. For 86% of the products, 
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the answer was "Yes," while for the remaining 14%, "Don't know" was the 
response. Independent of what the nominator indicated, some attempt 
was made to locate and identify the type of evidence for measured student 
gains, and classify the gains as either cognitive or affective. Such 
evidence was typically found in research reports or articles obtained 
from the developer of the product. The type of evidence varied from a 
tightly controlled experiment to recorded observations of behavior. 
Although it was found that 11% of the products had evidence of cognitive 
gains based upon the use of pre- and post-standardized tests with a con- 
trol group, evidence for cognitive gains was not located for 75% of the 
products since adequate documentation was not available. Similarly, for 
92% of the products evidence for sustained cognitive gains over time 
could not be located. Evidence of measured affective gains were located 
for only 16% of the products, while evidence of sustained affective gains 
was located for only 3% of the products. The data regarding evidence 
of cognitive and affective gains should be interpreted cautiously since 
they were based solely upon the reports and documents the developers 
elected to send. No attempt was made to initiate a thorough search since 
doing so was outside the scope of the study. 

ConUributions Toward Goals Through Facilitating Factors 

The nominator of each product was asked to indicate whether or not 
the product contributed to acceptable educational goals through one or 
more of the facilitating factors (i.e., organizational efficiency, class- 
room climate or operation, learning procedures or methodology, or per- 
ceptual-motor skills) . In addition, as with measured gains the project 
staff attempted independently to verify the contribution towards goals 
by reviewing documents supplied by the developers of the products. It 
may be recalled that in order to meet the effects criteria by which 
products were selected, a product had to have produced results which 
suggest a measurable gain, or to have contributed toward one of the 
facilitating factors. For most of the products the nominator didn’t 
know if the product had contributed to one of the facilitating factors. 
"Don’t know" was marked for 85% of the products for organizational 
efficiency, 67% for classroom climate or operation, 64% for learning pro- 
cedures or methodology, and 81% for perceptual-motor skills. Here, also, 
as with measured gains, there was very little evidence available in the 
documents in hand to support that the products so rated actually did 
contribute to the facilitating factors. 

Type of Developer 

Developer simply refers to the type of organization primarily 
responsible for the development of the product. About half of the pro- 
ducts (47%) were developed by government agencies or regional labora- 
tories and centers. About one-fourth (27%) were developed by private 
non-profit organizations, while the same percentage were developed by 
profit-making organizations. 

Table 12 shows the frequencies and percents of the 117 selected 
products by type of developer and by focus. Forty-one percent of the 
products had government as type of developer and a focus on subject 



matter. Somewhat less than a quarter of the products (23%) were developed 
by profit-making firms and had a focus on subject matter, while a similar 
percent of products had a focus on subject matter and were developed by 
private non-profit organizations. Only the private non-profit type of 
organization produced products that focused on organizational efficiency; 
these numbered only three. 



Table 13 shows the frequencies and percents of the 117 selected 
products by type of developer and by content. There was no consistent 
pattern across type of developer or across type of content, but there 
was some tendency for profit-making organizations to stress produces with 
a language content (15%) . Government groups seemed to have some ten- 
dency to stress science and social studies products (18% and 19% of the 
products, respectively), as did private non-profit organizations (13% 

and 15%) . 



Location of Developer 

Location of developer refers to the section of the United States 
where the product was developed. The sections or divisions were based 
upon the accepted U.S. Office of Education regional divisions of the 
United States. The frequencies and percents of the 117 selected pro- 
ducts by location of developer are shown in Table 14. Developing 
ganizations tended to be located more frequently in the northeast (54%) 
and west (30%). As seen in Table 15, while products developed in the 
west frequently tended to be products of government agencies or 
regional laboratories (19%), products developed in the northeast were 
about equally distributed among the three classifications of develop- 
ing organizations (16% government, 18% profit-making, and 20% private 
non-profit) . 



Characteristics of the 21 Selected Products 

Characteristics in Terms of the Additional S election Criteria 
and Constraints 



As noted in the Method section above, the following selection 
criteria and constraints were applied to the pool of 117 products to 
determine the final list of 21 exemplars: focus, format, grade level, 

extent of use, reported gain, visibility, and type of developer. 

Table 16 shows the distribution of the 21 products across these 
criteria and constraints. Nine products focus on language, seven on 
mathematics, seven on science, eight on social studies, three on voca- 
tional education, and four on one or more of the facilitating factors. 
Products were selected to represent both comprehensive formats with 
many types of media and limited formats. Seventeen products had high 
extent of use (i.e. , were in use in more than 100 schools), three 
moderate use (25 to 99) schools) , and one low extent of use (5 to 24 
schools). For evidence of student gain, 11 products were rated high, 
eight moderate, and two low. (See Method section "initial Selection 
of Exemplary Products" for basis used in rating gain.) Five products 



had high visibility and 16 moderate visibility as rated by the project 
staff. A government agency developed nine of the products, while 
seven were developed by private non-profit organizations, five by 
profit-making firms. (As noted on Table 16, the data reported in this 
paragraph are not necessarily consistent with subsequent tables since 
product selection was based upon information from nominators rather 
than results of the later case studies.) 

List of the 21 Exemplary Products 

The following is a list of the 21 products selected using the ad- 
ditional selection criteria and constraints. Their developers and the 
location of the developing organizations are also shown. These are the 
products for which complete case studies were prepared (see Appendix H) , 
and whose characteristics are reported below. 

1. Arithmetic Proficiency Training Program (APTP); Science Research 
Associates, Inc.; Chicago, Illinois. 

2. The Cluster Concept Program; The University of Maryland, Industrial 
Education Department; College Park, Maryland. 

3. The Creative Learning Group Drug Education Program; The Creative 
Learning Group; Cambridge, Massachusetts. 

4. Developmental Economic Education Program (DEEP); Joint Council on 
Economic Education, in cooperation with its Affiliated State 
Councils and Centers for Economic Education; New York, New York. 

5. Distar Instructional System; Siegfried Engelmann & Associates, 
University of Oregon; Eugene, Oregon. 

6. Facilitating Inquiry in the Classroom; Northwest Regional Educa- 
tional Laboratory; Portland, Oregon. 

7. First Year Communication Skills Program; Southwest Regional Labora- 
tory for Educational Research & Development; Inglewood, California. 

8. The Frostig Program for Perceptual-Motor Development; The Marianne 
Frostig Center of Educational Therapy; Los Angeles, California. 

9. Hawaii English Program; The Hawaii State. Department of Education 
and the University of Hawaii; Honolulu, Hawaii. 

10. Holt Social Studies Curriculum (Discovery Approach to American 
History); Carnegie Social Studies Curriculum Development Center, 
Carnegie-Mellon University; Pittsburgh, Pennsylvania. 

11. Individually Prescribed Instruction — Mathematics (IPI — MATH) ; 
Learning Research and Development Center, University of Pittsburgh, 
Pittsburgh, Pennsylvania; Research. for Better Schools, Philadelphia, 
Pennsylvania; Apple ton-Century.-Crof ts , Inc., New York, New York; 
Baldwin-I^ltehall School District, Pittsbur.gh, Pennsylvania. 



12. Intermediate Science Curriculum Study; The Florida State 
University, Intennediate Science Curriculum Study Project; 
Tallahassee, Florida. 

13. Materials and Activities for Teachers and Children — The MATCH 
Program; The Children's Museum; Boston, Massachusetts. 

14. Program for Learning in Accordance With Needs (PLAN); American 
Institutes for Research and Westinghouse Learning Corporation; 

Palo Alto, California. 

15. Science — A Process Approach; American Association for the 
Advancement of Science; Washington, D. C. 

16. Science Curriculum Improvement Study; Science Curriculum 
Improvement Study Project, University of California; Berkeley, 
California. 

17. Sesame Street; Children's Television Workshop; New York, New 
York. 

18. The Sullivan Reading Program; Sullivan Associates; Menlo Park, 
California. 

19. The Taba Social Studies Curriculum; The Taba Social Studies 
Curriculum Project, San Francisco State College; San Francisco, 
California. 

20. The Talking Typewriter or The Edison Responsive Environment 
Learning System; Thomas A. Edison Laboratory, a Subsidiary of 
McGraw Edison Company; Englewood Cliffs, New Jersey. 

21. Variable Modular Scheduling Via Computer; Stanford University 
and Educational Coordinates, Inc.; Sunnyvale, California. 

Brief narrative descriptions of these 21 products can be found in 
Appendix D. Appendix G contains a detailed description of the 21 
exemplary products; this description is in the form of a matrix of pro- 
ducts by the data elements used on the Product Data Fiecord (described 
in the Method section above) . The quantified characteristics of the 21 
exemplary products as described below are based upon data in Appendix G. 

Product Descriptions 



Product characteristics . This section includes information on 
type of developer, type of distributor, focus, grade level, and target 
population of the 21 exemplary products. The data u^re obtained 
primarily from the product developers during the case studies. There 
was more than one developer in some cases, although ordinarily one 
agency or organization alone was responsible for each of the products 
studied. Type of developer included: regional laboratory; R&D center; 



university; other government (which refers to county, state, public 
school district, or military agencies who may have been the primary 
developer) ; private non-profit (which refers to such groups as AIR, 
who have developed the product even though under government funding) ; 
and profit-making (which usually refers to conunercial publishers who 
are set up to make profits). 

Table 17 shows the frequencies and percents of the 21 exemplary 
products by each type of developer. Most frequently, universities, 
private non-profit organizations, and profit-making organizations were 
the developers (38%, 33%, and 29%, respectively). It is interesting to 
note that in no case did an organization classified other government 
(e.g., a school district) develop a product by itself. A third of the 
21 products were developed by two different types of organizations 
working together. In interpreting these results, it should be recalled 
that type of developer, as classified for the 117 products, was con- 
sidered in selecting the 21 exemplary products. 

Table 18 shows the frequencies and percents of the 21 exemplary 
products by type of disseminator. Type of disseminator refers to the 
kind of organization currently disseminating information about the pro- 
duct or intending to take this as a continuing responsibility. It does 
not refer to organizations who disseminated the product in the past and 
then quit. Dissemination refers to any of the channels by which infor- 
mation about the product and its use was brought to the attention of 
potential users, and not necessarily to sale of the product. For a 
third of the products there were more than one type of organization 
doing the dissemination. The dissemination of about 71% of the pro- 
ducts was by a profit-making organization; by Itself, 38%; with a 
university, 14%; or with a private non-profit organization, 19%. As 
might be expected, none of the 21 products were disseminated by R&D 
centers. Universities were disseminators only in partnership with a 
private profit-making organization (14%). One of the 21 products. 

The Cluster Concept Program, had no disseminator; 62% were disseminated 
by one organization, while 33% were disseminated by two organizations. 

Table 19 shows the frequencies and percents of the 21 exemplary 
products by type of focus. Focus refers to the major influences of 
the product as they affect students. Focus could be on one or more of 
the following: subject matter; organizational efficiency; classroom 

climate or operation; learning procedures or methodology; or per- 
ceptual-motor skills. Subject matter refers to language, mathematics, 
science, social studies, or vocational education. Organizational 
efficiency refers to ways of arranging the classroom, the schedules, 
time, space, and structure, etc. Classroom climate or operation refers 
to ways of changing operations or interactions among students or be- 
tween students and teacher within the classroom Itself. Learning pro- 
cedures and methodology include ways of presenting information or 
getting students to ask questions, etc. Perceptual— motor skills refers 
to a particular skill area. A prime example is probably that of the 
Frostig perceptual— motor skills program materials that teach visual and 
visually-oriented motor coordination. 
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It may be seen in Table 19 that about half of the 21 products 
focused only on subject matter, and about one-third on subject matter 
and learning procedures. No product focused only on classroom climate 
or operation, or only on learning procedures or methodology. However, 
38% of the products included learning procedures or methodology as a 
focus. Subject matter was included as a focus for 86% of the products. 

In interpreting the data on focus for the 21 products, it should 
be noted that focus was one of the items considered in selecting the 21 
products. It is, nonetheless, interesting to compare the results in 
Table 19 with Table 1, which shows focus of the 117 products. Both 
tables show similar percents (within 3%) for subject matter, organiza- 
tional efficiency, classroom climate, and perceptual-motor skills. 
However, for the 117 products, 10% had learning procedures or methodol- 
ogy as a focus, whereas for the 21 products 38% did. This may simply 
reflect the fact that project staff learned more regarding the details^ 
of the products studied than was available to the nominators of the li7 
products. 

Grade level of the 21 exemplary products is presented in Table 20. 
Grade level simply refers to the grades (i.e., K-12) in which the pro- 
duct is being used. The only products that were designed for a single 
grade level were for kindergarten. These were the First Year Communi- 
cation Skills Program and Sesame Street. Three products span kinder- 
partcn throuch grade 12; DEEP, the Hawaii ;:.nclish Program, and Vari- 
able Modular Scheduling. About three-fourths of the products were 
designed for grades 1 or 2, while only 29% were designed to include 
grade 10. One product, the Cluster Concept Program, was designed for 
grades 11 and 12 e-ly. The data in Table 20 differ, in a number of 
cases, from data on the 117 products shown in Tables 6 and 7. This 
may have been the result of having used grade level as one of the 
items considered in selecting the 21 products. 

The last product description is concerned with percent of target 
population. Percent of target population is the intended percentage 
of all students in the U.S. at a given grade level that a product is 
aiming for, as recorded by the product developer. Most of the pro- 
ducts (90%) investigated were intended for 100% of the target group. 
One product, Holt Social Studies Curriculum, is designed for average 
and college-bound high school students, or the upper 75% of the stu- 
dents. The Cluster Concept Program was designed for ^0% of grades 11 
and 12 students — those who typically would enroll in vocational edu- 
cation courses or in the general curriculum rather than in college 
preparatory courses, but who want some occupational preparation while 
remaining in the mainstream of the educational program. Generally, 
the percents of target population reported by the developers of the 21 
products were equal to or higher than those reported by the nominators 
of the 21 products, as may be seen by comparing Appendices E and G for 
the 21 products. (Data for all 117 products are shown in Table 8.) 

Objectives of product . The objectives of the product were the ob- 
jectives that the product was designed to help students or school per- 
sonnel achieve. Table 21 shows the frequencies and percents of the 21 
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exemplary products by the degree ot specificity with which the product 
developers laid out actual student performance objectives, as rated by 
the project staff. About two-thirds of the products had specific ob- 
jectives C29%) or vorv specific objectives (38%). Nineteen percent had 
somewhat specifieci objectives, and 14% general objectives. No product 
developer had specified only very general objectives. 



Description of materials . The format of a product refers to the 
mode in which information is presented; e.g., through textbooks, tele- 
vision, or laboratory equipment. A product might have one or more of 
the various types of formats. Table 22 shows the frequencies and per- 
cents of the 21 exemplary products for each type of format. All pro- 
ducts included some type of paper format such as textbooks, workbooks, 
manuals or guides, or tests. However, only three products (14%) used 
paper products only, hut none used only textbooks or only workbooks or 
only tests. Although about half of the products used an audio-visual 
format (i.e., television; films including slides, strips, and movie; 
and audio records and tapes), none used only this format. Sesame 
Street, for example, uses both television and manuals. Kits (charts, 
maps, models, games, or laboratory equipment) were used in 71% of the 
products, but always in conjunction with at least paper products. Over 
half of the products (52%) used displays such as charts, maps, and 
models. Three products (14%) used the computer: the Arithmetic Pro- 

ficiency Training Program, which also used paper products and kits; 

1..-.:. ri..\r', \ used paper products, nu.l i o-vi sual , and kits; 

and T.ii'Kinc Tvpowr i : . which in addition to tlic computer used paper 

products and aud i o-visual . The data in Table 22 are not comparable to 
those sho\,m in Table 5 for the 117 selected products. However, format 
was one of tlie items considered in selecting the 21 products from the 
pool of 117. 



Table 23 shows the frequencies and percents of the 21 exemplary 
products in terms of the format of materials used in training personnel. 
All products except one, the Creative Learning Group Drug Education 
Program, had teacher training materials, although this product does 
provide a teacher's manual for use in the classroom. Similar to Table 
22, almost all (95%) of the training materials included a paper format, 
and about half (5 7%) used an audiovisual format. However, only about 
a fourth used a kit. Tlie computer was used as a format tor presenting 
training in only one product. Project PLAN. All but nine products used 
more than one format, and eight (38%) used only paper products. All 
products but three used manuals or guides as at least one of the for- 
mats for teacher training materials. 



Subject matter content of a product, as shown in Table 24, refers 
r.-i ’-i-.ich of the five major content areas (i.e.. Language, mathematics, 
.science, social studies, or vocation education) the product was designed 
for. Only two of the products were designed for vocational education, 
the Cluster Concept Program and DEEP, and one of these (DEEP) in con- 
junction wich social studies. The percentage of products designed for 
the other four major content areas varied from about one— fourth to 
about two-fifths; 38% for language; 29% for mathematics, 24% for 




41 



science, and 33% for social studies. About 24% of the products were 
designed for multiple content areas, while two of the 21 exemplary 
products had no subject matter content. These two were Facilitating 
Inquiry in the Classroom and Variable Modular Scheduling. As mentioned 
above, subject matter content (in conjunction with focus) was one of 
the items considered in the selection of the 21 products. A comparison 
with Table 3, which shows subject matter content of the 117 selected 
products, reveals roughly similar percents for language, mathematics, 
and vocational education. However, the 21 products contained a lower 
proportion of the products with science or social studies content than 
the original pool (24% vs 39% for science, 33% vs 42% for social 
studies) . 

The cost of the product to the user refers to how much it costs the 
user per student per year, in dollars, to use tiie product; typically 
this means purchasing the product. In estimating the cost per student, 
members of the project staff considered the life of non-consumable 
materials to be five years. For materials which come as one per class, 
the cost was divided by 30 to arrive at the per pupil cost. If materials 
were designed to ba used on a sharing basis in a number of classrooms, 
realistic maximum utilization of materials was assumed. Table 25 shows 
the frequencies and percents of the 21 exemplary products by cost per 
student per yeair. For four of the products, it was not possible to 
determine the cost to the user. These products were the Arithmetic 
Proficiency Traininc Procram, Cluster Concept Program. DEEP, and the 
Taba Social Studies Curriculum. About half of the products cost the 
user five dollars or less per student per year. About 20% of the pro- 
ducts cost the user from $30 to $200 per student per year. Thus, the 
cost of the 21 exemplary products covered a very wide range, from $1 to 
$200 per student per year. Products costing $1 per student per year 
were Facilitating Inquiry in the Classroom, the MATCH Program, and 
Sesame Street. The Talking Typewriter is at the other end, at an 
estimated $200 per student per year. 

Period of student use . The period of student use was quantified as 
either the total number of hours within a semester during which the pro- 
duct is used, or, if the product is designed only for two or three weeks, 
the total number of hours for the product's application. In Table 26, 
the frequencies and percents of the 21 exemplary products by period of 
student use are shown. About one-third of the products were designed 
for continuous use for 20 to 50 hours per semester. About one-fourth 
were designed for continuous use for 80 to 130 hours per semester. 

About 14% of the products were designed for continuous use for more than 
130 hours per semester. The product with the longest continuous use, 
over 400 hours per semester, is Project PLAN. Only two of the products 
were designed for non-continuous use: the Arithmetic Proficiency 

Training Program and the Talking Typewriter (where the period of use 
depends upon the child). Period of student use was not applicable for 
three products: DEEP, Facilitating Inquiry in the Classroom, and 

Variable Modular Scheduling. 
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Other product descriptions . Another product descriptor is parent/ 
community involvement, which refers to whether the product provides iji 
its instructions for some parent or community involvement during the 
use of the product. Each product was classified by the staff using a 
five-point rating scale of no provisions, few provisions, some provxsions, 
many provisions, and extensive provisions. More than three-fourths of 
the products made no provisions or only a few prov'isions for parent/ 
community involvement. One product provided some provisions, four pro- 
ducts many provisions, and no product provided extensive provisions for 
such involvement. 

The degree of specificity of assessment measures was also con- 
sidered. It refers to the degree to which specific criteria were 
developed by the product developer to assess student achievement. This 
was usually related to the degree to which the developer specified stu- 
dent performance objectives. Products were classified on this variable 
by the project staff using a five-point scale of none, little, some, 
much, and very much. Over half of the products included assessment 
measures xvhich were highly specified. 

Product Origins 

Kev personnel. Key personnel were defined as those individuals 
whose involvement was critical to the development of the product, as 
judged by the project staff. As seen in Table 27, the number of key 
personnel ranged from one to 16. About two-thirds of the products had 
three to six kev personnel, while about one-fourth had seven to 16 key 
personnel. The’ remaining three products had only one or two key per- 
sonnel. Using the total cost of getting a product from its origin to 
the users as a general index of the size of a project, there does not 
appear to be any appreciable relationship between the size of a project 
and the number of key personnel. For the Sullivan Reading Program, 
three key personnel were identified} DEEP had four} Project PLAN had 
eight} Sesame Street lA; and the Hawaii English Program 16. The total 
costs for the Sullivan Reading Program and the Hawaii English Program 
were about the same. 

An attempt had been made to determine the number of years of 
experience of kev development staff, but sufficient information was not 
avaiJable. Table 28, however, shows the number of key development 
staff for the 21 exemplary products with doctorates. The number ranged 
from zero to 12. Sixty percent of the products had one to four key 
personnel with doctorates, while two products had no key personnel wxth 
doctorates. Nineteen percent of the products had five to seven key 
staff with doctorates, while two products, the Hawaii English Program 
and IPI, had 12. The percent of key staff that had doctorates ranged 
from zero to 100%, with an average of 60% over the 21 products. The 
products for which all key development staff had doctorates were the 
First Year Communications Skills Program with six, IPI with 12, 
Intermediate Science Curriculum Study with six, and Science A Process 
Approach with five. 



In Table 29 the composition of the key developmental staff for the 
21 exemplary products is presented. Staff specializations used were: 
teachers (elementary and secondary); college faculty members; adminis- 
trators; and R&D personnel not classifiable in one of the previous three 
categories. It may be seen that about three-fourths of the products 
were developed by a staff of more than one type. Seventy-one percent 
of the products had R&D staff as key personnel, while 57% of the pro- 
ducts had college faculty, and 52% administrators. Only two products. 
Facilitating Inquiry in the Classroom and Taba Social Studies Curricu- 
lum, had teachers as key personnel; this does not mean that teachers 
helped in the development of only two products, but that teachers were 
key personnel in only two products. None of the 21 products had only 
teachers or administrators as key staff. Four products (i.e., 19%), 
however, had only R&D personnel as key staff. The two products with 
only college faculty members as key staff were the Cluster Concept 
Program and the Sullivan Reading Program. 

When the data in Appendix G used to prepare Table 27 were compared 
with data used for Table 29, it was found over the 21 products that on 
the average 2% of key staff were teachers, 37% college faculty, 11% 
administrators, and 49% R&D personnel. 

Sources of ideas for product . Data for the 21 exemplary products 
were collected related to the determination of user needs; that is, how 
the developer actuallv went about determining tiic needs of the user for 
a product before, it w.^s developed. Table 30 sho\;.s the frequencies and 
percents of the 21 exemplary products for each of four methods of 
determining user needs. For all products, user needs were assessed by 
at .least making an educated guess based on past experience. Know- 
ledgeable people were asked or the literature reviewed to determine 
user needs for 85% of the products. For 43%, a sample of potential 
users was observed or asked; while for 24% of the products, performance 
measures were used. The most frequently used combination of methods 
was the educated guess plus asking knowledgeable people or reviewing 
the literature (43%). For 19% of the products all four methods of 
determining user needs were used. Those products were the Frostig 
Program for Perceptual-Motor Development, the Hax^;aii English Program, 
Science Curriculum Improvement Study, and Sesame Street. User needs 
were never determined solely on the basis of asking knowledgeable 
people, observing user samples, or performance measures. As mentioned, 
there was always an educated guess based on past experiences. 

Funding for product . About 48% of the products were developed with 
funds from the government, 24% with private funds, and 28% with both 
government and private funds. The frequencies and percents of the 21 
exemplary products by total cost to get the product from origin to 
user are shown in Table 31. Total costs ranged from $50,000 for the 
Creative Learning Group Drug Education Program to $14,000,000 for 
Sesame Street. About half of the products had cost $2 million or more 
each by the time they reached the user. About one-fourth of the pro- 
ducts cost less than a half million dollars by the time they reached 
the user. For most of the products these total costs could not be 
broken down by the developers into costs for development, evaluation. 





3 nd d if £us ion j so b r^ost brsfllcdown could not bs msds. No cosb informs” 
tion was available for three products: the Distar Instructional’ Sys- 

tem; Frostig Program for Perceptual-Motor Development; and IPI. 



Product Development 

Number of organizations that participated in product de velopment. 
In determining the number of organizations that participated in product 
development, only those organizations that were responsible for 
development of the product and made major contributions in terms of 
actual development activities were included. For 574 of the products 
there was just one organization involved in the development; for 33% 
there were two organizations; for one product, the Sullivan Reading 
Program, there were three; and for DEEP there were four. The percent 
of each organization's resources used for product development was 
initially a matter of interest but was later omitted since the informa- 
tion was not usually available from the product developers. 



Actual procedures for development of product . During the case 
studies, the number of formative evaluation cycles taken for each pro- 
duct wav noted. The number of formative evaluation cycles refers to 
the number of attempts to try out the product, obtain information, and 
then use that information to make revisions in the product, thereby 
producing a new phase or model of the product. Of the 21 exemplary 
products, 29% had one formative evaluation cycle, 19% had two cycles, 
29% had three cycles, and 24% had four or more cycles. Concerning the 
stage at which formative evalution was initiated, as rated by the staff 
on a five-point scale, none of the developers of the 21 products 
started formative evaluations very late, 5% (or one product. Cluster 
Concept Program) started late, 14% started midway, 48% started early, 
and 33% started very early. 

The developers employed various procedures or combinations of 
procedures during formative evaluation. The procedures used for the 
first cycle of formative evaluation for the 21 exemplary products are 
shown in Table 32. Most of the product developers (91%) used multiple 
procedures during this cycle. The most frequently used procedure was 
"development staff observed in use" (91%) , w'hile the least frequently 
used procedure was ''development staff taught using the product 
(43%). For 29% of the products, all four procedures were used: the 

two just mentioned plus "asked classroom teacher" and "obtained per- 
foirmance measures." These latter two procedures were used for 81% and 
67% of the products, respectively. The procedures "asked classroom 
teacher" and "obtained performance measures" were never used alone. 



Similar inform.iticn is found for the second cycle of formative 
‘valuation in Table 33. The trends remained approximately the same, 
2 xcept that asking the classrcom teacher was the most frequently used 
jrocedure. It was used for all 15 products that had a second cycle 
jf formative evaluation. For 27% of the products with a second cycle, 
lil four procedures were used. Similar information is found in Table 34 
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for the third and subsequent cycles . The trends remained the same as 
for the second cycle for the 11 products with a third cycle. However, 
during the fourth and subsequent cycles, there was a tendency for the 
four procedures to be used at a somewhat more equal frequency for the 
five products involved, although developmental staff taught using only 
two of the products; 

Generally, the same procedures were employed across the cycles of 
formative evaluation. Six products had only one cycle. For the 15 
that had a second, 12 used the same procedures as for the first cycle, 
two dropped "development staff taught," and one dropped this procedure 
and "asked teacher." Considering the 11 products that had a third 
cycle, nine used the same procedures employed for the second cycle, 
while two dropped "development staff taught." For the five products 
that had more than three cycles, four used the same procedures as for 
the third cycle, while for one product "obtained performance mea^-rures" 
was substituted for "staff observed in use." 

Table 35 shows the frequencies and percents of the exemplary pro- 
ducts by the. degree to which formative evalution data were used to 
modify the products, in the opinion of the staff. After the first 
cycle, the degree of modification dropped considerably. During cycle 
one, 67% of the products were modified very much, 14% much, 14% some, 
5% (one product. Facilitating Inquiry in the Classroom) only a little, 
and no products not at all. For the 15 products with a second cycle, 
one. Cluster Concept Program, was not modified at all; three were 
modified some; seven modified much; and four modified very much. The 
majority of products with a third cycle were modified some: none were 
not modified or modified only a little, while two were modified very 
much. Of the five products with more than three formative evaluation 
cycles, two were modified only a little or some; two were modified 
much; and one, the Frostig Program for Perceptual-Motor Development, 
was modified very much. 

Summative Evaluation 



Large and small scale field tests . The results given in this 
section were based upon reports of studies supplied by either the pro- 
duct developers or disseminators. They were at times difficult :o 
evaluate and compare since the information reported varied in detail. 

No attempt was made to review the general professional literature or 
search other sources since doing so would have been outside the scope 
of the study. Field tests refer to evaluations conducted after a 
product was developed to determine its effectiveness. Small scale 
field tests were defined as those with fewer than 100 students, large 
scale fie’ld tests those with 100 or more students. 

Table 36 shows the frequencies and percents of the 21 products by 
number of field tests conducted. About half of the products that were 
field tested were field tested only once or twice, and 24% of the pro- 
ducts (five products) were not field tested. Several products were 
field tested three or more times, and reports on six field tests of the 



Distar Instructional System were available. Table 3tS also shows the 
number of small scale and large scale field tests, in addition to the 
total number of field tests. Fifty-seven percent of the products had 
no small scale field tests, 38% no large scale field tests. The 
Science Curriculum Improvement Study had five small scale field tests, 
while the Sullivan Reading Program had four large scale field tests. 

Table 37 shows the frequencies and percents of the 21 exemplary 
products by type of funding source for the major field test for each 
product. The major field test for a product was defined as that field 
test with the largest number of students, whether or not it was 
classified as large or small. Funding sources used in Table 37 are: 
regional laboratory, R&D center, university, other government (e.g., 
state or local), private non-profit, and profit-making. About half of 
the products which were field tested received funding for the major 
field test from multiple funding sources. Other government funds sup- 
ported major field tests for 48% of the products. 

In comparing the funding source for the major field test with type 
of developer (as representing funding source for product development), 
it was noted that for five products the funding souce was the same for 
both product development and major field test. For four products, other 
government funds (i.e., state or school district) supplemented the 
sources used for product development for conducting the major field 
test. In the case of two products, one developed by a private non- 
profit organization and the other by a profit-making firm, the major 
field tests were conducted using only other government funds. These 
products were, respectively, the MATCH Program and the Talking Type- 
writer. Product development for IPI was supported by both regional 
laboratory and R&D center funds, while regional laboratory funds 
only supported the major field test. For Science — A Process 
Approach, regional laboratory funds were used to support the major 
field test, while the product was developed by a private non-profit 
organization (AAAS). The Distar Instructional System was developed 
using private non-profit and university funds, whereas the major 
field test was funded by other government and profit-making organi- 
zations. The tendency for other government funds to support field 
tests is apparent. 

As seen in Table 38, the geographical extent of all field tests 
conducted for the products ranged from one to 11 states, with most 
products being tested in one to four states. Project PLAN was field 
tested in 11 states. Information concerning th'j number of schools 
participating in all field tests for a product was available for only 
eight products. The range in number of schools was one to 40 schools; 
no one number of schools tended to dominate. As shown in Appendix G, 
information concerning the number of students in all field tests for 
a product was available for 12 of the 16 products which were field 
tested. The range in number of students was 15 to 28,000 for Science — 

A Process Approach. However, half of the 12 products were tested with 
1,000 or fewer students. Either one or two organizations participated 
in the field tests; for 52% of the products, one organization 



participated. An attempt was made to determine the number of field 
test staff used in the major field test for each product, but infor- 
mation was available for only three products. Similarly, data on the 
cost nf the major field test were available for only one product. 

Product effectiveness . During the case HL^aies, an attempt was 
made to collect information regarding evidence for measured cognitive 
and affective student gains resulting from the use of the product, and 
classes of evidence of sustained cognitive or affective gains. Since 
the reports available to the staff varied so widely in content and 
detail, it was impos.'iible to locate sufficient specific information to 
make quantifying the results meaningful. As a rtsult, data in Appendix 
G is limited to indicating the number of studies summarized in the 
Product Development Reports by type of gain or effect. These data are 
shown in Table 39. Positive gains means the experimental group met the 
criteria set by the product developer, or the group performed signi- 
ficantly better than the control group. Mixed gains means the experi- 
mental group met some of the criteria set by the product developer but 
not all of the criteria, or the group performed significantly better 
than the control group on some of the subscales of a test, but not all 
of the subscales. Negative gains means the experimental group did not 
perform significantly better than the control group. Data are shown 
for studies of product effectiveness conducted during product develop- 
ment (hence including formative evaluation studies if they had evidence 
on product offccti vcness) nnd for large and small scale summative eval- 
uation studies conducted after the product was developed. 

As shown in Table 39, for a majority of the products (67% to 86%), 
there were no studies available that indicated positive cognitive gains. 
Considering studies conducted during product development, for six pro- 
ducts positive cognitive gains were reported in one study, and for one 
product (Distar Instructional System) such gains were reported in two 
studies. Four products had positive cognitive gains reported in one 
large scale field test, one product had them reported in two large 
scale field tests, and two products (Science— A Process Approach and 
the Sullivan Reading Program) had them reported in three such field 
tests. Only three products had positive cognitive gains reported from 
small scale field tests: for one product they were reported in one 

study, and for two products they were reported in two studies. Mixed 
cognitive gains were reported: in one or two studies conducted during 

product development for four products; in reports of large scale field 
tests for seven products; and in reports of small scale field tests for 
one product. There were no studies conducted during product develop- 
ment that indicated negative cognitive gains, but such evidence was 
reported from a large scale field test of one product and from a small 
scale field test of four products. 

Few studies reported affective gains. Positive affective gains 
were reported: in one study conducted during product development for 

the Intermediate Science Curriculum Study: in one large scale field 
test for IPI, the MATCH Program, and Taba Social Studies Curriculum; 

in one small scale field test for two products. The Science 
Curriculum Improvement Study had positive affective gains reported 
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in two small scale field tests. Mixed affective gains were reported 
in two studies conducted during product development of the Cluster 
Concept Program. 

Even fewer studies reported evidence of facilitating effects: 
that is, that a product contributed towards accepted educational goals 
through school organizational efficiency, classroom climate or opera- 
tion, learning procedures or methodology, or improved perceptual-motor 
skills. Positive facilitating effects were reported in one study con- 
ducted during product development of both the Frostig Program for 
Perceptual-Motor Development and Variable Modular Scheduling. Positive 
facilitating effects were also reported for a large scale field test 
of Facilitating Inquiry in the Classroom. 

In considering results on product effectiveness, the question 
arises as to whether or not a product is supposed to show cognitive or 
affective gains. Performing such an analysis would have required 
classifying each of the goals of all 21 exemplary products, which was 
not feasible from the information available. In addition, as noted 
above, data on effectiveness were obtained from reports available to 
the staff, which did not necessarily cover all of the goals of the 
products. In many cases studies of affective gains are made for pro- 
ducts primarily oriented towards producing cognitive gains. For ex- 
ample, a product may have teaching mathematics as its major purpose, 
but a study could also have been done to see whether or not a student s 
attitude tov^ards mathematics changes as a result of using the product. 

As an alternative to classifying products in terms of their goals, 
evidence of effectiveness was considered in terms of the focus of the 
products: whether or not they focused on subject matter only, the 

facilitating factors only, or on both subject matter and facilitating 
factors. These data are given in Table 40, which shows evidence of 
effectiveness by focus for studies conducted during product develop- 
ment, large scale field tests, and small scale field tests. In Table 
40, when more than one study was available for a product, only the one 
with the most positive evidence was used. For example, if three 
studies were , available for a product showing positive, mixed, and 
negative gains, respectively, the positive results only would have been 
considered. Data are also sho\-m in Table 40 considering the most 
positive evidence for each product over all studies of it, regardless 
of whether the study was conducted during product development or as a 
large or small scale field test. 

It may be seen in Table 40 that there were seven products with 
focus on subject matter and facilitating factors. Considering studies 
conducted during product development, one of these products. Talking 
Typcv'L'itor , produced positive cognitive gains. The Intermediate 
Science Curriculum Study produced both positive cognitive and positive 
affective gains. Mixed cognitive gains were found for two products, 
while for three products there were no reports available regarding 
studies on effectiveness conducted during product development. Two of 
the seven products, DEEP and Talking Typewriter, showed positive 
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cognitive gains in large scale field tests; Project PLAN showed mixed 
cognitive gains; IPI mixed cognitive and positive affective gains; 
and the MATCH Program positive affective gains. Considering the most 
positive evidence over all studies for these seven products: the 

Arithmetic Proficiency Training Program, DEEP, and the Talking Type- 
writer produced positive cognitive gains; the Intermediate Science 
Curriculum Study produced positive cognitive and positive affective 
gains; IPI and Project PLAN resulted in mixed cognitive and positive 
affective gains; and the MATCH Program produced positive affective 
gains. There were not reports available that indicated studies of 
facilitating effects for any of the seven products with focus on 
subject matter and the facilitating factors. 

Table 40 also shows that there were eleven products with focus on 
subject matter only. For six of them gains were reported in studies 
conducted during product development. One of these, Cluster Concept: 
Program, produced mixed cognitive and mixed affective gains, while the 
following five produced positive cognitive gains: Distar, First Year 

Communication Skills Program, Holt Social Studies Curriculum, Science — 

A Process Approach, and the Sullivan Reading Program. Six of the eleven 
products with focus on subject matter only had evidence of product 
effectiveness from large scale field tests: Distar, Hawaii English 

Program, Science — A Process Approach, Sesame Street, and Sullivan 
Reading resulted in positive cognitive gains, while for Taba Social 
Studies Curriculum there were mixed cognitive and positive affective 
gains. Either positive or mixed cognitive or affective gains were 
reported for four of the five products with focus on subject matter 
when small scale field tests were considered, while for one product 
negative cognitive gains were noted. (However, this product had pro- 
duced positive cognitive gains in a large scale field test.) When 
considering the most positive evidence over all studies for the eleven 
products with focus on subject matter: Distar, First Year Communication 

Skills, Hawaii English, Holt Social Studies Curriculum, Science — A 
Process Approach, and Sesame Street produced positive gains; Science 
Curriculum Improvement Study and Sullivan Reading showed positive cogni- 
tive and positive affective gains; Taba Social Studies Curriculum result- 
ed in mixed cognitive and positive affective gains; and Cluster Concept 
Program produced mixed cognitive and mixed affective gains. No studies 
of product effectiveness were available for the Cluster Concept Program. 
Finally, as noted in Table 40, there were three products with focus on 
only one or more of the facilitating factors. Considering the most 
positive evidence over all studies, all three produced positive facili- 
tating effects. Facilitating Inquiry in the Classroom showed these 
effects in a large scale field test. The Frostig Perceptual-Motor Skills 
Program demonstrated positive facilitating effects in a study conducted 
during product development, as did Variable Modular Scheduling. In 
generally evaluating the data in Table 40, excluding products with a 
focus on the facilitating factors only, there is no apparent relationr- 
ship between the focus of a product and the type of evidence (cognitive 
or affective) of effectiveness. 

Extent to .which products were modified based upon field test results . 
The extent to* which the 21 exemplary products were modified as a result 



of the major field test of each product (that is, the field test with 
the largest sample size) was rated by the staff on a five point scale 
as shown in Table 41. The judgment could not be made for two products. 
Of the remaining 14 products that had field tests, the Science Curri- 
culum Improvement Study was not modified at all as a result of its 
major field test. Seven products were modified only a little, four 
products somewhat, and two products much. These two products were 
the MATCH Program and Taba Social Studies Curriculum. No products were 
modified very much as a result of the major field tests. 

Diffusion 



Agency participation and diffusion efforts . The marketer or dis- 
tributor of a product may or may not be the same as, the disseminator. 

In this study marketer was taken to refer to the organization currently 
doing the selling. Usually this was only one organization, although in 
a few cases two organizations were concurrently marketing the same 
product . 

Table 42 shows the frequencies and percents of the 21 exemplary 
products by type of marketer or distributor. (Type of disseminator is 
shown in Table 18.) As expected, 86% of the products were being 
marketed by profit-making organizations, usually commercial publishers, 
but in conjunction with another type of organization for 14% of the pro- 
ducts. One product, Cluster Concept Program, had no marketer or dis- 
tirbutor. For 81% of the products the primary developer was currently 
disseminating, while for 48% of the products the primary developer was 
currently handling the marketing. 

Diffusion strategy . Table 43 shows the frequencies and percents of 
the 21 products by type of technique used for dissusion, and when during 
the development of the product the technique was used. When the tech- 
nique was used- was rated by the staff on a five point scale of very 
late, late, midway, early, and very early. Diffusion techniques were 
classified as tell, show, involve, train, and intervene. The diffusion 
techniques used are nonexclusive, and one product could use all of them. 
Tell refers to any sort of publication, special paper read, word of 
mouth, etc. Show usually refers to a demonstration or display. In- 
volve means to have the user participate in some stage of the actual 
development of the product. Train means to instruct the user about the 
use of the product in the classroom through, for example, workshops 
or institutes. Intervene means to exert pressure from levels of 
authority higher than the user to adopt the product. As seen in Table 
43, no diffusion technique was used very late in the development cycle. 
Inteirvene was used for only three products: early for Hawaii English, 

midway for IPI, and late for Sullivan Reading. Tell, show, and train 
were used more frequently. Tell was used late for one product (Cluster 
Concept Program), midway for five products (24%), early for nine pro- 
ducts (43%), and very early for six products (29%). Show was generally 
used later in the development cycle than tell: late for two products 

(10%), midway for seven (33%), early for nine (43%), and very early for 
two. Only one product. Cluster Concept Program, did not use show. 

For four products train was not used as a diffusion technique. It was 



used late for four products (19%), midway and early foi’ six products 
(29%, respectively), and very early for Science — A Process Approach. 
Involve was used for 14 products, having been used midway for the 
Science Curriculum Improvement Study, early for seven products (33%) , 
and very early for six products (29%). In summary, except for inter- 
vene most techniques were used midway to very early, with tell and 
involve being used very early for 29% of the products. 

Factors affecting diffusion . Some of the factors affecting 
diffusion included: the availability of competitive alternative pro- 

ducts; the degree of change (on the affective level) required in using 
the product; compatibility with other school practices; and divisi- 
bility of the product. The availability of competitive alternatives 
refers to whether there are other products in the same content area 
or with similar approach or foci for roughly the same grade levels. 

For 33% of the products there was no other competitive alternative 
available. For 29% there were one or two alternatives available, for 
33% there were three to five alternatives, while for one product, the 
Sullivan Reading Program, there were six to nine alternatives. 

Degree of change required refers to how many modifications have 
to be made in the classroom organization or staff attitudes and 
functioning in order to use the product. As rated by the staff: 

Sesame Street required no change on the affective level; 52% of the 
products required little change; 24% required some change; the Cluster 
Concept Program required much change; and very much was required for 
IPI, Project PLAN, and Variable Modular Scheduling. 

Compatibility with other school practices was rated by the staff 
on a four point scale from very compatible to very incompatible. Very 
incompatible meant that the product conflicts with other ways of doing 
things in school, probably with teacher approaches and values. A com- 
bination of variable scheduling and student selected topics in sex and 
drug emanation would be very incompatible to most systems. At the 
other end of the extreme very compatible meant that use of product re- 
quires no real change in classroom organization, structure, or opera- 
tion. This would occur, for example, if the product was only a slightly 
different set of materials on a typical topic in a subject area. Three 
products (14%) were rated as very incompatible with other school 
practices. These three were Cluster Concept Program, Project PLAN, and 
Variable Modular Scheduling. Two products (10%) were rated as incom- 
patible: the Hawaii English Program and IPI. Eleven products (52%) 

were rated as compatible, while five were rated very compatible. 

Data were also available about the divisibility of the products, 
which refers to whether components of the product could be purchased 
independently and used separately. Eighty-six percent of the products 
had components that were designed to be used independently, and 81% 
had components that could be purchased independently. 

Adoption 

Extent bf product use . Extent of product use simply refers to the 
number of schools and the number of students using the product, and the 
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number of states where the product is being used. Information on the 
number of schools using the product was available for only nine pro- 
ducts; the range in number of schools for these products was 8 to 
35,000, this latter for the Sullivan Reading Program. Information on 
the number of students using the product was available for only 15 
products. The range of number of students for these products was 
7,000 to 7,000,000 students. The Science Curriculum Improvement Study 
is reportedly in use by a million students. Sesame Street and the 
Sullivan Reading Program by five million. The estimated seven million 
for DEEP is an overestimate; it is the total number of students in the 
school districts that. have adopted DEEP, the actual number of students 
using che product being unknown. The number of states in which each 
product is used was known for only 15 products. It ranged from one 
state for the Rawaii English Program to all 50 states for the Inter- 
mediate Science Curriculum Study, Sesame Street, and the Sullivan 
Reading Program . 

Installation procedures . Installation procedures that affect 
product adoption focused on: the need for special facilities and 

equipment; the need for classroom modification; the extent of product 
modification allowed; the need for special teacher training; special 
teacher training provided; extra staff requirements; the degree of 
administrative support needed; and the importance of public relations. 
The need for special facilities and equipment refers to physical things 
that might have to be bought through the marketer or otherwise in order 
to use the product. About 76% of the products did not require special 
facilities and equipment, in the judgment of the staff. The need for 
classroom modification refers to any reorganization required from the 
basic 1-30 teacher-centered operation and schedule. This variable was 
rated by the staff on a five point scale. No classroom modifications 
were required in the use of the Creative Learning Group Drug Education 
Program or the First Year Communication Skills Program. Few classroom 
modi.f ications were needed for 13 products; some modifications for two 
products; many modificatons for Cluster Concept Program, IPI, and 
Variable Modular Scheduling; and very many modifications for Project 
PLAN. 



The extent of product modification intended by the developer 
refers to the extent to which the developer intended to allow the 
school or teacher to change the product to suit their own needs. Some 
products were made to be used in a very flexible fashion, while others 
were pretty much self-contained and inviolable. As rated by the staff, 
no product modification was intended by the developers of Distar, the 
Hawaii English Program, and Sesame Street. Little modification was 
allowed for 10 products (48%), some modification for five. Much modi- 
fication was allowed for DEEP and Variable Modular Scheduling, while 
very much was allowed for Facilitating Inquiry in the Classroom. 

The need for special teacher training recognized or specified by 
the developer was also rated by the staff. In part, the need was 
inferred by the developer's insistance on teacher training. In cases 
in which the developer recommended it but sold the product irrespective 
of teacher training, only some need was indicated. No need was a case 
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in which supposedly a self-instructional manual was sold or given to 
the teacher as part of the product, and no other training was 
indicated. All products required at least a little need for teacher 
training. Four (19%) required little, seven (33%) required some, four 
required much, and six (29%) required very much. These six were: 
Facilitating Inquiry In the Classroom; Frostig Pcrcoptual-Motor Skills 
Program; Hawaii English Program; IPI; Project PLAN; and the Taba Social 
Studies Curriculum. 

The availability of teacher training services which the developer 
was willing and able to provide was also considered. At least a 
little teacher training was provided for all products. A little 
teacher training was provided for four products (19%), some was pro- 
vided for 11 products (52%), and much was provided for four products. 
Complete teacher training was provided for Facilitating Inquiry In the 
Classroom, and for IPI. 

Table 44 shows the frequencies and percents of the 21 products by 
type extra staff requirements. That is, the kind of extra staff 
required in order to efficiently use the product. Over half of the 
product' did not htive extra staff requirements. The use of four 
required extra supervision (that is, school district personnel such as 
supervising teacher or assistant supervisor), five required the use of 
paraprof essionals such as teacher's aids, two required additional 
teachers, and four required the use of consultants from outside the 
school district. One product. Talking Typewriter, required the use of 
all four types of extra staff. 

The staff rated each product on the degree of administrative 
support required, i.e. , how much effort the superintendent and principal 
primarily must exert to install and see that product is successfully 
adopted. All products required at least a little support. Thirty- 
eight percent needed little support, 33% needed some, and 14% needed 
much. The three products needing very much administrative support were 
Project PLAN, Talking Typewriter, and Variable Modular Scheduling. 

The importance of public relations prior to the adoption of each 
product was rated as minimally important, important, or critical. The 
rating was based upon the degree to which certain issues are currently 
sensitive with the public. Public relations prior to adoption were 
considered minimally important for 24% of the products, important for 
57%, and critical for 19%. The four products for which it was judged 
critical were: Hawaii English Program; IPI; Talking Typewriter; and 

Variable Modular Scheduling. 

Feedback from users . Table 45 shows the frequencies and percents 
of the 21 products by method employed by the developmental organization, 
.the dissemination organization, and the marketing organization for 
getting information back from people who have adopted the product. For 
most, of - the products (91%) informal collection procedures were used, 
and it was the only procedure used for 71% of the products. For 24% 
of the products systematic sampling was used. A survey of all uses was 
employed for only one product, the Hawaii English Program, which also 
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used the other two methods of obtaining feedback. No methods were used 
for the Cluster Concept Program. 

Future of the Products 




Product developers were asked the amount of expected use by 1973, 
in terms of students. This number could be estimated for only 10 of 
the 21 products. Estimates for these 10 products varied from 30,000 
to 10,000,000 students. The developers of the First Year Communica- 
tion Skills Program anticipate one million students by 1973, while for 
Science Curriculum Improvement Study two million are expected. Ten 
million students are expected to use Sesame Street and the Sullivan 
Reading Program. 
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DISCUSSION AND CONCLUSIONS 



The reader who has survived the forty-odd tables in the preceding 
Results section may need a breath of re-introduction to this project 
and its limitations. The project consists of 21 case studies. The 
cases were deliberately selected so that subject matter, grade level, 
and objectives varied among the 2T (to mention a few dimensions; the 
multiple criteria applied to select the cases are described above and 
shown in Table 16). The 21 cases, as a set, are in no way a random 
sample of educational product populations. The sampling was intention- 
ally biased, via the successive criteria, and the small total number 
of cases was spread across a variety of product categories. General- 
ize tions from such a sample must be handled with care. 

The individual case studies, although published separately as 
Product Development Reports, are an integral part of this report. 

Each of the 21 Product Development Reports may, however, be read inde- 
pendently. Each represents an empirical base from which hypotheses 
and tentative generalizations, including and beyond those contained in 
the present report, may be drawn. The overall project was designed to 
provide a basis for more definitive studies. The detailed product 
development histories represent a necessary and significant contribu” 
tion toward developing more accurate predictions of the potential 
impact of educational products. The project staff found no comparable 
case studies. Nor have others who searched the field. Havelock (1971) 
in his extensive review of innovation in education concludes that the 
first of five major needs in the area is that "We need more case 
studies which carefully document and report . . ." (Pp . 11-12) Within 
this perspective some issues and findings of the current project can 
be reviewed. 



Problems Encountered in the Procedures of the Study 

The relative effectiveness of various search procedures used to 
identify the pool of potential products and the final products for 
case study was reviewed in the Results section above. Once products 
were selected, two sources of information were explored: reviewing 

documentation of the product's history, and interviewing staff who had 
participated in the development. It was found that most organizations 
had bulky files of documentation. However, gaps and omissions in both 
records and memory were frequently encountered. Few organizations were 
able to provide a detailed breakdown of product development /diffusion 
costs. Frequently, only gross figures summarizing their organization's 
yearly budget, or the total of awarded contracts or grants were avail- 
able. Cost accounting for the various phases of product development is 
an area for which records are acutely needed. 

Commercial publishers tended to be more restrained about giving 
out information than were staff from universities, research and devel- 
opment centers, regional laboratories, school districts, etc. There 
seems to have been no established history of open and complete communi- 





cation regarding product development on the part of commercial publish- 
ers • However, to some extent this may reflect naivety on our part 
inasmuch as the project staff lacked experience in the commercial 
publishing business and tended to have considerable history of inter- 
action with universities, school districts, regional laboratories, etc. 

For several of the early case studies difficulty was experienced 
in obtaining sufficient documentation in advance so that the initial 
documentation would be useful to the site team. Rec^uests for documen- 
tation were made at least 60-90 days in advance of possible visitation 
dates. Frequently, such advanced documentation was simply a packet of 
bj^ochure , non— critical publicity handouts. These were useless. When 
adequate previsit documentation was not available, site visit teams 
were handicapped in interviews and wasted a good deal of time in ex- 
tracting information that could have been more economically gathered 
through reviewing reports. Frequently, the needed documentation was 
available in the organization's files. In fact, this was usually the 
case. However, a number of factors mitigated against its being readily 
sent: sometimes the documentatidh was one only and the organization 

did not wish to spend the reporduction costs; information needed was 
scattered throughout a number of documents which required an inordinate 
mailing weight; or the knowledgeable staff had moved on to other pro- 
jects and were not in close touch with the documentary repository. 

The case study procedure used required extensive on-site inter- 
views with the key personnel who had been involved in the development 
of each product. The 21 exemplary product studies included samples 
from various organizations across the United States. In most cases, 
the product had been completed from one to three years previously. 
Arranging for economically feasible site visits , that is within budget 
constraints, required continuous schedule juggling: scheduling within 
a given organization, among the various staff members, and between 
different development organizations. Typically, the development staff 
of a given product were involved in other projects at the time of the 
site interview. They were frequently working for other organizations 
and in other locations. There were several cases in which the key 
developer, whose interview was crucial, was working in an entirely 
different region of the United States. 

The schedule did not allow sufficient time and resources for pro- 
longed on— site review of records . Site visitors occupied by far the 
bulk of their time in interviews with key staff members. Many organi- 
zations had large files of supporting documentation. However, little 
time was available to peruse these thoroughly. In general, project 
staff had to be satisfied wi';h an overview of the documents, taking 
copies of those which could be released from the files for careful 
review at a later time. 

A consistent attempt was made during the preparation of every Pro- 
duct Development Report to communicate with the product staff regarding 
the fidelity and completeness of the report draft. Academic schedules, 
vacations, and even job changes during this period caused some diffi- 
culties* However, each report was reviewed by either the key staff who 
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had been interviewed, or by a staff member designated by the organi~ 
zation. In all cases the draft report appeared to have captured the 
major events and problems in the product's developmental history as 
perceived by the developer. Suggestions tended to focus on appro- 
priate credit to personnel, on toning down a few pungent phrases, and 
generally on what could be considered details of report. Each draft 
report was approved by the developer. All developer suggestions 
related to factual levels of descriptions were incorporated in the 
final drafts. Suggestions regarding interpretations were given 
serious consideration. 



Possible Biases in the Information Collected 

The case studies are based primarily upon information collected 
from the development /diffusion agencies. All the information was 
checked as closely as time and availability of records would permit. 
However, no corresponding amount of effort was devoted to a search 
among users. In several cases, extensive user information was 
uncovered. However, this was serendipitous. Some mitigation against 
the possibility that positive claims about the products have been 
overblown by developers' reports was the general evidence collected 
from records of formative evaluations and field tests. 

The study was essentially a retrospective historical search. 

As such, it is subject to all the biases inherent in attempts to 
reconstruct the past. In all case studies the staff v;as able to 
interview several respondents independently. Cross-checks of infor- 
mation revealed few discrepancies, although each member of the 
development team tended to present different emphases, depending on 
the interest and special experience of the respondent. There were 
few instances in which respondents' recollections were refuted by 
records. Typically, the respondent described the general character- 
istics of a particular procedure on a product's history and referred 
the project staff to documentation for detail. 

There had been some initial concern, and this was emphasized by 
the consultants to the project, that the type of interview schedule 
used could color the results. Early in the project a rather tight 
interview schedule was proposed which tended to follow a systems 
development model of product development and diffusion. This highly 
structured schedule was rejected and an open-ended procedure substi- 
tuted. Tests of the structured schedule with AIR staff, who had 
experience in product development, showed that respondents gave pro- 
fessionally desirable answers, i.e. , every history followed a tidy 
systems development sequence. All interviewers were trained on the 
outline for Product Development Reports contained in Appendix F. 
However, this was used primarily as a checklist before and after each 
interview. 
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Factors Related to Product Development 



Four factors related to the development of the 21 exemplary pro- 
ducts warrant special discussion based upon a review of the individual 
Product Development Reports. These are key personnel, funding pat- 
terns, time required for development, and formative evaluation. 

Key Personnel 



The most noticeable attribute of the personnel most directly 
responsible for the development of the 21 products is the staff mix: 
a mix of staff experience and professional specialities. This mix is 
evident in two ways. Over the longer term his'tory of a project there 
was typically more scientific or research orientation to the earlier 
investigations than found later. University backgrounds were pre- 
dominant. Later development efforts by professionals with an instru- 
ctional technology or systems development background built upon these 
earlier ideas and findings. This point will be discussed further in 
the section on funding. 

In the actual development of a product, a mix of professional 
specialities was usually the case. This is true even in the smaller 
scale projects in which only, for example, three key development 
staff were identified. The minimal mix appeared to be: content 

skills often combined with instructional development skills; evalu- 
ation and research background sometimes combined with systems 
development skills; and a user, practical orientation. For example, 
for Facilitating Inquiry in the Classroom, the working team of 
three consisted of an experienced inquiry trainer — content, a 
research and evaluation specialist, and a participant observer who 
initiated user groups during tryouts. This team was supported by a 
larger group of varied specialists within the development organiza- 
tion. The Sullivan Reading Program's team consisted of a professor 
of languages who had spent years learning how to develop programmed 
materials, a psychologist, and a linguist who could write. 

There were no one man products among the 21. Often a key mover 
or driving force could be identified: Joan Cooney in Sesame Street, 

John Flanagan in Project PLAN, Marianne Frostig in the Frostig 
Perceptual Motor Skills Program, Alvin Karplus in the Science Curri- 
culum Improvement Study, and many others. They initiated, sustained 
the project through lean times, and fought for support. As the 
product was built, these key movers often assumed a major management 
role. But the development was accomplished by teams — either small 
or large . 

It should be noted that the Results section indicates a product 
with one key staff member. This was reported for the Creative Learn- 
ing Group Drug Education Program. The organization is relatively 
small. The key mover, a very forceful personality, played a major 
role in all steps of development.. However, a team of several members 
with backgrounds in programmed learning and teaching produced the 
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materials. Similarly, a key staff team of two reported for the 
Talking Typewriter refers to only the hardware component of the pro- 
duct. The MATCH Program is reported having a key staff of two. There 
were two movers and originators in this museum based project. However, 
the MATCH boxes or units were each developed by a separate team, a 
team of varying backgrounds. 

A recurring theme among the small group teams was the difficulty 
and importance of effective sustained cooperation. The successes 
were seen, mixed teams that had completed the product. The failures 
were buried in old personnel records. For almost every product there 
were casualty lists of staff departures. Beneath the more acceptable 
reasons for leaving, the AIR site visitors typically found an in- 
ability to work with other members. In some cases delays in pro- 
duction could be directly traced to staff discord, resolved by 
departure of some, followed by replacements. Effective solutions 
appeared to occur througli trial and error, or error and fire. The 
surviving teams worked together continuously, had frequent informal 
contacts with one another, and learned a fair amount about each other's 
discipline, i.e., its approaches, problems, and requirements. However, 
the AIR site visitors were not geared to systematically identify and 
explore these group interactions in more depth. 

Access to a wider variety of professional input on a part-time 
base was typical. Large organizations supplied these in-house. Small 
organizations obtained consultant or part-time services from individ- 
uals in other organizations. 

The role of teaching and teachers in the development of the 21 
products was paradoxical. The products are designed for use by 
teachers. Yet teachers themselves are not well represented in the key 
development staff. A number of "reformed teachers" who had become 
administrators, curriculum advisors, or college staff served on 
development teams. In several of the larger scale projects, summer 
writing conferences were employed in which groups of teachers helped 
to write materials. Teachers and classrooms served an active role 
throughout most products in giving feedback to prototype versions. In 
Project PLAN and Science — A Process Approach, for example, teachers 
assisted in both preparing materials and tryouts, but they were not 
classified as key development personnel for the purposes of this 
study. The most extensive role played by teachers was in the early 
version of the Taba Social Studies Curriculum. Local district staff 
selected the topic and the project leader, Hilda Taba. Dr. Taba 
shaped the direction of the project with the assistance of local 
administrative and curriculum staff, but teachers constructed almost 
all components of the early product. In the later version of the 
Taba product, teachers played a more typical and restricted role. 

On the other hand, key staff who had no previous public school 
background frequently plunged into classrooms with prototype products. 
Edwin Fenton of Carnegie Institute of Technology taught versions of 
the Holt Social Studies Curriculum to many junior high classes. Dr. 
Karplus, Professor-of Physics at Berkeley, personally taught science 
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units in elementary grades for several years to forge the Science 
Curriculum Improvement Study. Siegfried Engelmann worked through 
version after version of what is now Distar in face to face teaching 
with disadvantaged children in several states. Other products show 
similar involvement with teaching-learning. For approximately half 
the products supporting staff working with teachers conducted the 
early explorations in the classroom. 

Funding Patterns 




I 



Most products were directly supported wholly or in large part by 
government sources. The striking success of private enterprise 
typically occurred when it built on prior development funds supported 
by federal sources. Science Research Associates invested in Distar 
following a series of Office of Education investments in preceding 
versions. Apple ton-Century has played an important role in the IPI 
materials based upon prior and continuing federal support. Private 
monies entered into the development of other products and into the 
packaging and marketing of almost all. 

No government funds were directly involved in several products. 

The Creative Learning Group Drug Education Program was built with a 
shoestring of private backing by a staff claiming first hand know- 
ledge of the topic. In order to bring in desperately needed resources, 
this product was marketed as a completed product before it was 
developed. Dr. Pros tig has literally spent a lifetime developi.ng the 
products associated with her name. Her "kitchen table" students are 
now adults. These were students needing remedial instruction whom 
Dr. Frostig taught in her kitchen — having no laboratories or classrooms. 

In almost all cases of large scale product development, prior 
work which piloted content, format, method, etc. was utilized. The 
developers of the Taba Social Studies Curriculum had already done low 
cost tryouts of their work; Bereither, assisted by Engelmann, had 
developed and tried the predecessor of Distar; and Suchman had been 
supported in a series of small scale projects for the development 
preceding Facilitating Inquiry in the; Classroom. These were pri- 
marily government funded efforts. 

Time Required For Development 

The 21 products reflect a relatively long period of development. 
There are difficulties in many caSes in determining when development 
on the current product started, as opposed to work on a forerunner. 

A three to five year period of development is typical — exluding time 
on the prior foundation and/or product. A number of products reveal 
a five year history from 1966 to 1971 — a fertile era, the beginning 
of which coincides with the initial operational year of the regional 
laboratories. The Arithmetic Proficiency Training Program, the com- 
puter based arithmetic program, required five years of intensive 
work by Science Research Associates. Distar appears to have been 
underway in the early '60's — a nine year period. The Intermediate 
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Science Curriculum Study required approximately six years, while 
Science — A Process Approach took from 1961 to 1968. Massive appli 
cations of resources do not appear to shorten development under a 
five year period — witness the Hawaii English Program and the First 
Year Communication Skills Program. 

Formative Evaluation 



The formative evaluation cycles for earh product, their number 
and timing, are presented in Tables 32, 33, and 34. The major infer- 
ence from these tables is that the 21 products used from one to four 
formative evaluation cycles. They did have formative evaluations. 

The extent of these evaluations was, however, considerably greater 
than what is indicated by the tables. The site visitors recorded the 
number of "reported formative evaluations." They sought and identi- 
fied evidence of relatively complete reports~something akin to a 
weak, journal article. There was in addition to these, although in- 
adequately documented and reported, an almost continuous evaluation 
that occurred in most of the products. Typically, components of the 
product were worked through with target audiences, and development, 
staff taught, participated, or observed. Records of these _ activities 
were kept in a loose fashion. Reviews of the tryout were held almost 
immediately afterwards. Development staff and participating teachers 
reviewed problems, apparent achievements, etc., and identified areas 
for modification. Documentation was of the hip pocket variety, and 
the evaluation did not emerge later as an Identifiable cycle. 

For example, the Science Curriculum Improvement Study staff both 
taught and worked alongside teachers for years doing this sort of 
tryout and loose evaluation. No one kept a record of the replications 
of materials tried out and results observed in the several classrooms 
of the Frostig Learning Center. Fred Newton tried out early versions 
of the Facilitating Inquiry in the Classroom materials in workshop 
after workshop, making it difficult to clearly identify the formative 
evaluation cycles. IPI-Math imdergoes a continuous evaluation process 
through the Oakleaf School, then to the demonstration schools, and on 
to the pilot schools in the program. The reporting strategy used in 
trying to quantify the number of formative evaluation cycles tended 
to underestimate these efforts. In many cases the evidence on which 
revisions could be made was based on inferences by the development 
staff, and student performance measures were sketchy. Products which 
are committed to clear overt performance measures tended to use them 
in this process, e.g-. , the Arithmetic Proficiency Training Program, 

IPI, and Project PLAN. Sesame Street placed major emphasis on the 
attention getting power of its prototypes. Innumerable sessions were 
conducted measuring the attentiveness of small children to the screen, 
and attractive distractors were introduced to measure the holding power 
of the program. Little of this series of informal experiments emerged 
as repeated cycles. A recent report concluded that current educa- 
tional products, receive, little or no evaluation before being placed on 
the market (Education USA, 1971). For the 21 products, this was not 
the case . 



Features of the 21 Exemplary Products 



Outstanding Attributes 

A number of tl)c 21 products presented unique or outstanding 
attributes that , in the judgment of the AIR staff , should be reported 
as useful potential strategies for other product developments. It 
should be emphasized that the following are illustrative. The con- 
cerned reader should consult the individual Product Development 
Reports for a complete account of each product’s attributes. Further- 
more, these are subjective judgments. 

1. The Arithmetic Proficiency Training Program presented a clear 
strategy of analyzing a series of skills into a hierarchy of 
sub-skill components, and of arranging materials to teach 
these skills so that each learner can start at his own level 
and receive immediate and appropriate feedback. 

2. DEEP has a decentralized development system which is indi- 
vidualized for each user group. An emphasis on assisting 
schools on the process of change has been developed into well 
defined procedures. 

3. Distar has developed a procedure for tryout and revision 
closciv roiipled with teacher training f.iiat appears both 
different and strong. 

4. The gradual revision of Facilitating Inquiry in the Classroom 
from a complex and relatively obscure master-disciple intern- 
ship to an explicit system that can be taught by persons of 
relatively low experience may be generalized to a variety of 
products . 

5. The Frostig Perceptual-Motor Skills Program revealed an 
effective and sustained involvement with the target audience 
in a setting that supported experimentation and revision of 
materials . 

6. The First Year Communication Skills Program illustrates the 
effective application of instructional technology — a systems 
development model supported by adequate resources. 

7. The Hawaii English Program presents a mobilization of 
resources within a state to solve a major problem — a massive 
effort demanding involved cooperation of all educational 
ins ti tut ions . 

8. IPI — Math has developed a hierarchical network of supporting 
and participating schools, a linkage system that can be 
generalized and exported to other areas. 



9. The Intermediate Science Curriculum Study carefully planned 
a diffusion sequence that was coordinated with, and helped 
to shape, product development from its initial stages • 

10. The MATCH boxes show a way to package and deliver beauty 
and cultural heritage. 

11. The potential scope of individualization of instruction 
using available materials is shown by Project PLAN. The 
development of specific performance objectives, accompanied 
by achievement measures, across all subject areas reveals 
what can be done. 

12. Sesame Street has led the field for educational television 
in ways that are obvious to most readers. Their intensive 
promotion-diffusion effort is not as well know and scrutiny 
of it will be rewarding. 

13. Science — ^A Process Approach offers useful guidelines on the 
mobilization of a scientific community to determine the 
needs for and feasibility of product development. 

14. The Science Curriculum Improvement Study reflects the 
effect of direct involvement of development staff in early 
classroom tryouts. The target audience substantially 
shaped the development of the product . 

15. The Sullivan Reading Program presents an array of diffusion 
approaches, allowing numerous options to potential users. 

Weaknesses of the Products 

This section again is based upon the subjective judgments of the 
AIR staff. The alleged weaknesses are presented, not as criticisms 
of any product or organization, but as potential red flags for other 
product developers. 

1. The lack of carefully planned diffusion strategies to 
accompany product development was evident in the history of 
a substantial number of the products. "Diffusion occurs 
after development and is largely planned afterwards" sum- 
marizes this approach. The, contrast with products whose 
diffusion had been made part of the original development 
plans and had been coordinated with development is obvious 
in the case studies. 

2. The development of high cost systems based upon special 
equipment not ordinarily found in schools has resulted in 
problems for several products. More involvement of 
potential users and sensitiveness to user Constraints 
appears called for in planning and development. 
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3. Several products ran aground when the developer began making 
arrangements for marketing. Units had been hand built in 
development. In small numbers, the expense of individual 
units had not totaled to a large sum. Subsequently, it was 
found that the materials were not suitable for low-cost mass 
production. The alternatives were substantial changes which 
may affect the quality of the product, or price the product 
off the general market. 

4. A number of products were completed without assessment tech- 
niques for users. As demand became evident, assessment 
devices were added. These post hoc techniques appeared of 
little substance or usefulness. 

5. In several large scale efforts substantial resources were 
expanded in early development without any apparent tryout 
and feedback. Tryouts were conducted, but after many months 
and dollars had gone into development. Results of the try- 
outs pointed to costly revisions. 

6. The lack of field tests for a number of products has been 
noted in the Results section. Provisions for field tests 
were not Included in plans or funding. As the products 
reached completion, the need to make them available tended 
to overcome any perceived need for field testing — unless it 
had been provided for in earlier phases of development. 



Products With Attenuated Impact 



If the product sample for this study had been sufficiently large, 
some failures might have been included along with the apparent 
successes. What factors operate in the case of products that met 
criteria, at one time at least, but now show signs of losing impact or 
not attaining potential? There is some evidence that a sample of such 
products is included in the project. 

The Cluster Concept Program is a vocational education product 
developed primarily by university staff in a department of industrial 
education. The products’s developmental history is similar to that 
of others in the sample with one prominent exception — no arrangements 
had been made, nor resources allocated, for diffusion. When the pro- 
duct was complsted and had been evaluated for effectiveness, the 
developers searched for diffusion support. In vain. No support was 
located and no useful base for diffusion had been developed previously. 
The project appears to be headed for a long dry storage. Actually, 
several other products made relatively late arrangements for diffusion, 
but the others were able to generate rescuing support. 

The Talking Typewriter presents a cyclical history. The original 
product, developed by Omar K. Moore in the late ’50’s, received broad 
dissemination coverage but was never developed for general use. Moore 
used it for his own research on learning. Later the product, or 
rather the hardware, was refined substantially by the present developers. 
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It was completed and made available for general use without software. 
Initial efforts met with no success. The product appeared destined for 
the shelf. An allied organization was formed to select and develop 
appropriate software. After a period, adoptions began to increase. 

The product now appears to be making at least a moderate impact. 

The MATCH Program, the contribution of the Children's Museum in 
Boston, captured the hearts of the AIR site visitation staff. Where 
else can a child worship at a Japanese altar, complete with ancestor 
table tj or use real knuckle bones in the manner of the ancient 
Greeks Yet, the MATCH boxes, the self-contained multi-media units, 
may be destined for a local role only. The materials, which include 
Grecian urns, Eskimo boots, mounted birds, and falconry equipment, 
have not lent themselves to commercial manufacture. Some authenticity 
had to be dropped, e.g. , the prototype version used real knuckle bones 
obtained from a local slaughterhouse and then boiled in a staff 
member's kitchen. Circulation of the MATCH boxes to Boston schools 
is brisk. Sales to schools at $500 per kit and interest by other 
museums has been low. Diffusion strategies were never given a sub- 
stantial place in the development. After the boxes were developed^ 
to meet the exacting standards of the developer, diffusion activities 
and commercial linkages were initiated unfortunately, perhaps too 
late . 



Generalizations of Findings to Current Hypotheses 
Regarding the Process of Innovation and Change 

The burgeoning literature on diffusion and change does not include 
any proportionately growing amount of case materials. It was hoped 
that the present 21 empirically based case studies would shed some 
light upon current hypotheses concerning innovation and change pro- 
cesses. It is felt that they do. The reader is cautioned that the 
selection criteria used, the search strategies employed to identify 
products, and the actual procedures in collecting case study informa 
tion may have resulted in a non-representative sample of innovative 
products. Furthermore, the information available from the Product 
Development Reports does not cover all of the major current hypotheses 
on innovation and change. It was not possible to get evidence bearing 
on a number of these hypotheses. In addition, relevant evidence is 
frequently of a low grade character. Often it represents a retro- 
spective estimate by the developer and/or the AIR project staff and is 

tangently supported by a documented or hard data. However, keeping 
these serious limitations in mind, some of the major notions related 
to the change process can be examined in light of what these 21 case 
studies do indicate. 

Hypotheses Regarding the Characteristics of an Innovation 

Those characteristics of an innovation associated with its likeli- 
hood of adoption which are most often repeated in the literature include 
compatibility, complexity, divisibility, trialability, and relative 



advantage (Roger?, 1962; Rogers & Shoemaker, 1971; BumeLt, 1953; 
Glaser, 1967; Havelock, 1971). 



Compatibility . Compatibility has been defined bv Rogers (1962) 
and Rogers & Shoemaker (1971) as "the degree to v;hich an innovation 
is consistent with existing values and past experiences of the 
adopters." The hypothesis is that the more compatible the innovation, 
the more likely it is to be adopted. This compatability , according 
to the hypothesis, may be located in the ' innovation s degree of con~ 
sistency with the potential adopter's values or his needs, or in his 
past experience and/or previously adopted ideas. As thus stated, the 
hypothesis provides an extensive range within which to locate points 
of fit or non-fit. Although often categorized as an attribute of the 
innovation itself, compatibility actually resides in the perceptions 
of the potential adopting audience — according to these who espouse 
the hypothesis. User perceived con^jatibility is the concept. 



Evidence cited for the hypothesis is usually a description of an 
innovation that failed ; upon post hoc analysis the innovation then 
appears to have been Incompatible with one or more user values or^^ 
experiences. One wonders what a similar search among successful 
innovations would uncover. Evidence regarding the 21 products is less 
than firm inasmuch as no direct measures of user perceptions were in- 
cluded in the study. No clear cut dramatic case of adoption failing 
because of a value contradiction, as often cited in ; he cross-cultural 
literature oii adoption, was found. 



Signs of value conflict were occasionally uncovered. For example. 
Distar's highly structured teaching is antithetical to many teachers. 
The programmed text approach of Sullivan Reading was at least formerly 
if not currently in conflict with some views on teaching. A number of 
the products were based upon a student inquiry, self-directed learning 
approach. Although this approach may be valued in the abstract, 
teacher training experiences indicated that much of what is entailed 
is not compatible with what is valued in practice by many users. The 
open discussions emphasized by the Creative Learning Group Drug Edu- 
cation Program appear to be uncomfortable, if not incompatible, to a 
number of users. The MATCH Program has emphasized nonverbal learning 
and has encountered signs of value priority differences, if not in- 
compatability. Comprehensive approaches to individualization appear 
to be applauded as a concept, but require administrative and teaching 
behavior changes that are not altogether congruent with existing 
patterns . 



As previously indicated, the current case studies were unable to 
obtain systematic or large scale data from adopters. Thus, on degree of 
compatibility the products were judged by the developer/disseminator 
staff interviewed and by the AIR site visitors. The most readily 
available indicator of compatability was the degree of change required 
by the product — organizational change or change required in staff 
behavior. The change may vary from a simple addition to a re- 
orientation of basic values and approaches . 
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The 21 products appear to cover a broad range of compatability on 
this measure. Variable Modular Scheduling is not compatible with the 
current daily class pattern of most schools. It requires extensive 
changes by administration and staff. The complete individualization 
of instruction required by Project PLAN and IPI ate quite incompatible 
with usual practices. To a lesser degree a number of the products 
demand extensive behavior changes from teachers. Facilitating Inquiry 
in the Classroom, Holt Social Studies Curriculum, and Science A 
Process Approach are examples of products that require teacher skills 
that must be provided by special training. Approximately 80% of the 
products are accompanied by more or less required specialized teacher 
training. The other 20% either recommended or offered optional 
training. 

On this evidence, which points more towards in comp stability with 
experience than values, the products cannot be regarded as highly 
compatible with current education. They reflect a tendency to forge 
ahead, or diverge from the existing, and to have procedures designed 
to bring the asers along. 

Complexity . "Complexity is the degree the innovation is difficult 
to understand and use" (Rogers, 1962). The hypothesis is that the 
complexity of an innovation is negatively related to its adoption 
(Rogers, 1962} Rogers & Shoemaker, 1971). At least three aspects of 
complexity have been delineated; the number of components in the pro- 
duct; the number of skills required before adoption; and the number 
of procedures required for maintenance. ' Complexity appears to be some- 
what confounded with compatability. A product may be incompatible by 
requiring skills beyond those presently exercised by adopter staff. 

The requirement of training to develop these skills adds to its 
complexity. 

Specialized teacher training is perhaps the most consistent factor 
contributing to complexity of the 21 products. All products required 
or recommended some teacher training. This has been discussed under 
Compatability. 

The number of components identified within each product varied 
extensively. In the more complex products, e.g.. Project PLAN, the 
Hawaii English Program, and IPI, AIR staff did not enumerate each 
specific component on a detailed level. The number appear to be awe- 
some. Installation of the Talking Typewriter and the Arithmetic 
Proficiency? Training Program require highly complex man-machine 
arrangements. Approximately one-third of the products are relatively 
complex in both number of components and installation procedures. DEEP 
requires an extensive planning activity involving the entire school 
district. On the other extreme. Sesame Street requires a TV set. A 
number of the profiucts require only a student and teacher text for 
their minimal version. Holt Social Studies Curriculum and the Inter- 
mediate Scd.ence Curriculum were intentionally designed to present a 
low complexity profile to users. In a similar fashion to their 
relationship to compatability, the 21 products extend across a broad 
range of complexity. 
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Divisibility . The hypothesis is that the more divisible a product 
the more probable is its adoption. Divisibility may refer to any of 
several attributes of the innovation. The following have been classi- 
fied as examples of divisibility: opportunity to try the innovation 

before adoption, being able to adopt the innovation piecemeal rather 
than all or none, small scale adoption, and divisibility of decisions 
to adopt in which group consensus is not required for adoption. Such 
attributes are sometimes labeled "trialability" (Rogers, 1969), 
emphasizing the capability of a trial run or a partial or tentative 
adop tion . 

Opportunities for a "hands on" trial, a workshop demonstration, 
firsthand observation of the product in use, etc. are specifically 
provided for a majority of the 21 products. The Arithmetic Proficiency 
Training Program, whose computer requirements make it indivisible in 
other respects, has emphasized demonstration tryouts of the product. 
Thousands of sample units of the Holt Social Studies Curriculum were 
distributed. The large scale products such as Project PLAN, DEEP, 
and the $40,000 Project "Read" off the Sullivan Reading Program dis- 
played marked efforts to involve potential users in the design of 
custom built versions. 

Approximately 80% of the 21 products are divisible in regard to 
the purchase of separate components. The Tabc Social Studies Curricu- 
lum can be purchased as simply as one teacher guide. From this, the 
range extends all the way to Variable Modular Scheduling and The 
Talking Typewriter which represent an "all or none" product. 

Divisibility in the sense of number of decisions required for 
adoption appears closely related to cost and complexity. Several 
products required administrative commitment. In others, the cost, 
ranging from $3 for one manual to over $5,000 for a complete system, 
would appear to predict the extent of decisions required. Project 
PLAN, which individualizes most subject areas, and Variable Modular 
Scheduling require behavior changes by a majority of staff. Pre- 
sumably a consensus decision is more likely. 

In summary, a majority of the products appear to be divisible 
in one or more senses of the term. 

Relative Advantage . "Relative advantage refers to the degree to 
which an innovation is perceived as being better than the idea it 
supersedes" (Rogers; 1962). The hypothesis is that the perceived 
relative advantage of the innovation is positively related to its 
adoption. Two aspects of relative advantage are stressed economic 
advantage and social reward. The term is most frequently used to 
denote relative cost. The hypothesis is that lower adoption and 
maintenance costs increase the likelihood of the adoption of the 
innovation. * 

The cost of installation of the 21 products presented a broad 
base of variance. Variable Modular Scheduling costs several thousand 
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to install. Project PLAN, The Talking T’rpewriter, the Arithmetic 
Proficiency Training Program, DEEP, and the Hawaii English Program 
present similar high exists. The installation costs ranged downward 
from these to $100 or less per classroom for Facilitating Inquiry 
in the Classroom, First Year Communication Skills, and the Frostig 
Program for Perceptual-Motor Development. The 21 products do not 
present a consistent picture of initial low cost advantages. 

Detailed information on maintenance costs was not obtained. 

The "social reward" meaning of relative advantage is used to 
include decrease of discomfort, increase of status approval by others, 
and expected pleasure. Any or all of those, as perceived by potential 
users, constitutes relative advantage. Little evidence was found on 
which to gauge this sense of the hypothesis. All of the products 
appeared to the AIR staff to possess one or more of such relative 
advantages. The hypothesis is so multi-faceted it may well be im- 
permeable to most evidence. And, as noted previously, the project 
did not include measures of the perceptions or attitudes of users. 

Additional Hypotheses Suggested by Havelock 

The above hypotheses are frequently encountered in the literature. 
In Havelock's study of innovation and change (Havelock, 1971), several 
additional hypotheses are proposed. 

Linkage . Linkage refers to the degree of interpersonal or inter- 
group connections and communication between developer/diffuser and 
user. The hypothesis runs that the more linkages that exist and the 
stronger the linkages, the more effective will be the utilization of 
knowledge. The hypothesis suggests that a variety of reciprocal and 
collaborative relationships between developer and user Increase the 
likelihood of adoption. 

The majority of the 21 development organizations show such link- 
age. They demonstrated a history over a sustained period of time of 
a wide variety of relatively close contacts with user representatives. 
Most of the development/diffusion organizations involved were con- 
cerned with other products and/or services to users. They had 
established a multilinear network of various channels of communication. 
Regional laboratories perhaps present the clearest example of this. 
Products eminating from regional laboratories tended to be disseminated 
against a prior background which usually included: regional need 

assessment; user representatives on advisory boards or directorates; 
many informal contacts and visits by both groups over a period of 
time; prior informative training workshops offered by the developer/ 
diffuser to user groups, etc. IPI and DEEP present examples of pro- 
ducts in which the linkage system between developer and the network of 
participating schools for the particular product is elaborate and 
strong. The interested reader is advised to peruse these case reports. 
Distar, with a network of demonstration, sites tied in to the product, 
reflects strong linkages. In several cases such as Sesame Street, 
linkage was built along with the product. The development of the 
Sesame Street home viewing groups and tutoring programs has established 
relatively firm ties with user groups. 



A few products revealed weak linkages. The Cluster Concept Pro- 
gram appeared to have developed very moderate tie-ins. The Talking 
Typewriter started with almost no-:, but has been devoting major effort 
to establishing such linkages after the product was completed. 

In the judgment of the AIR staff* the majority of the 21 products 
reflect moderate to strong linkages to their intended users. 

St ructure . From the developer/diffuser perspective structure 
refers to the degree of systematic organization within the group. 

Three aspects of structure are included! a division of labor; a co- 
herent view of the user system; and a planned strategy of diffusion. 

The hypothesis is that the more structure displayed by the developer/ 
diffuser, the more profitable is the adoption of the innovation. 

With respect to the first meaning of structure — organized division 
of labor within the development group — the 21 products present strong 
support. The specialty mix of staff teams has been discussed previously. 
In all cases a number of roles and professional specializations were 
organized into the production unit. Most of tlie groups had relatively 
large in— house ranges of specialization which were systematically brought 
to bear upon the product being developed. 

With respect to planned strategy for diffusion, the products clearly 
vary. Some started such planning early in development; others commenced 
at field test phases, or later. There were indications that this 
hypothesis may be confirmed. Although several products which made a 
flying leap at diffusion appear to be gaining adoption, the several 
"attenuated impact" products fall in this category. All products for 
which very early and extensive diffusion planning was found, and these 
were only four or five, showed solid adoption totals. This should be 
distinguished from sensitivity to user needs. Product developers 
frequently displayed signs of openness to user needs which shaped the 
decision to initiate development. However, little effort was then 
expended in planning eventual realization of the forthcoming product. 

Capacity . This concept refers to a conglomerate of power, sophis- 
tication, and status: indications that the developer/diffuser has high 

capabilities. The hypothesis is that the more capacity displayed by 
the developer, and presumably as perceived by users, the more likely 
is adoption. 

The large majority of developing organizations manifest relatively 
high capacity. The R&D centers, regional laboratories, the American 
Association for the Advancement of Science, etc. reflect capacity and 
there seems little doubt they would be so perceived by user groups. 

Only one product appears to emanate from a relatively low capacity 
group. 

Scientific Status. This . factor has been treated with caution in 
the literature (Havelock, 1971). The hypothesis is that the higher 
the scientific status of the innovation the more probable its adoption. 
Questions as to the degree of rationality in user decisions coupled 



with evidence that many mistrust a heavy emphasis on science have led 
to questions of the importance of this factor. 

Apparently the 21 developers of the products had similar questions. 
Scientific status of the products rests largely upon the reputation of 
individuals and the developing institution* the extent of systematic 
effort devoted to development, and a wealth of informal evidence 
accumulated during tryouts and formative evaluations. As a set, the 21 
products do not present a consistently strong position on controlled 
evidence supporting intended learner gains. Nor does such evidence 
appear to be a major feature of messages from developers to users, 
Demonstrated use in relevant settings and objectives were emphasized, 
not experimental evidence or field test results. 

Problems in Assessing The Change Hypotheses 

It appears that, when pressured, the hypotheses relating innovation 
characteristics to adoptic .1 retreat to the eye of the beholder. This 
study collected no direct evidence regarding user perceptions. Presum- 
ably, users might either perceive a product as having characteristics 
that the product lacked, or might fail to perceive existing character- 
istics of the product. Several of the hypotheses are a bit loose for 
testing. Some contain such multiple disjunctive concepts in their 
description of the attributes of an innovation that one would be pressed 
to find an instance not included. 

The variables included in the several change hypotheses are pre- 
sumed to interact with one another. Each empirical case study should 
be examined for the simultaneous effect all variables operating, 
which was outside the scope of the current project. Complexity effects 
may be balanced by divisibility, increased by low capacity, etc. 

Rogers & Shoemaker (1971) lists approximately 100 generalizations of 
approximately the same degree of precision as those discussed above, 
each containing one or more variables from the diffusion literature. 
Their simultaneous application to one case may present a problem of 
interest to others. 

It is probably quite obvious that the developer/dif fuser may com- 
pensate for a hindering attribute of his product by increased efforts 
stressing other aspects. Thus, the indications that a substantial 
number of the 21 products were relatively complex may be interpreted 
as "if such a product is to have substantial impact, this disadvantage 
must be overcome by compensating attributes or diffusion efforts." 

The argument is plausible. Sustained energetically it offers the 
hypotheses virtual immunity from all evidence currently available. 

The case studies were compared as a set against each of several 
change hypotheses. The 21 cases are in many ways a heterogeneous 
set. They do not differ neatly along any single dimension with other 
features held tightly in order. They are cases, not experimental 
treatments.. 



Conclusions That Appear Supported By the Study 



The following propositions appear to have moderate support from 
the results of the present study. The basis for each proposition has 
been identified in the Results section or previously in the 
Discussion. 

1. A variety of small scale developmental projects involving 
instructional innovations of an experimental nature provide 
a basis for later, more complete product development. 

2. A relatively lengthy and sustained application of resources 
and talent is required for product development. 

3. Product development is primarily the work of a staff team 
representing diverse disciplines and specialization. 

4. Formative evaluation initiated early in product development 
and applied almost continuously was evident in the products 
studied. 

5. Continuous open communication with user representatives is 
needed throughout the development of educational products. 

6. Substantial resources allocated early in product development 
to the design and development of diffusion strategies are 
required for effective diffusion later. 

7. A number of products did not plan for the design of appro- 
priate measures of effectiveness or make specific provisions 
for conducting product effectiveness tests. These activities 
must be provided for early in the development period if any 
scientifically reputable evidence of product effectiveness 

Is to be obtained. 



Identification of Ways Whereby Information on the 
Impact of Educational Products May Be Obtained 

In this section ways whereby information on the impact of educa- 
tional products may be obtained are noted. Several current systems 
present information across a wide range of both developing and avail- 
able products, including information relative to product impact. 
Through the use of such systems the Office of Education might contin- 
uously monitor the impact of educational research and development 
products . 



1. The ALERT (Alternatives for Learning through Educational 
Research and Technology) system of the Far West Regional Educational 
Laboratory was noted in the Method section above. ALERT focuses on 
products that have been tested and can demonstrate their effective- 
ness. However, to keep the system as relevant as possible, other 
types of programs are also being included; "maverick" programs that 



offer new, unique solutions to primary problems but have not been 
tested and evaluated; programs that are unique in their coverage of 
content; and programs that have been widely adopted but whose effect 
is not yet well known. ALERT attempts to provide sufficient informa- 
tion for the user to make a decision to adopt, reject, or adapt a 
development. 

There are several information sources available from ALERT. One 
is the "ALERT Recipe Box" which contains punched cards, each repre- 
senting a curriculum program, organizational arrangement, or other 
program relevant to schools’ needs. The cards are edge-punched accord- 
ing to the program's characteristic features. Basic facts about each 
program are printed on one side of the card; they include the program 
name, type, length of use, cost, name and address of developer, etc. 

An abstract, focusing on the content of the program and classroom 
activities, is printed on the other side. The purpose of the Recipe 
Box is to produce a rough identification for further exploration. 

Other ALERT information sources include: summaries including brochures, 

folders, and photographs; audiovisual briefings which show the program 
in action; and reports describing the goals, objectives, content, 
materials, teaching strategies, implementation requirements, costs, and 
evaluation of each program. 

2. As also noted in the Method section above, the Information 
Office for the network of R&D centers and educational laboratories pub- 
lishes the CEDaR (£ouncil for _Educational D^evelopment, and Research) 
catalog of selected educational development and research programs, pro- 
jects, and products. The catalog distributed in September 1971 contains 
summaries of programs listed individually on program sheets. Each pro- 
gram sheet shows the laboratory or center name, the program name, a 
program abstract, and a list of related projects for which project 
sheets are provided. The project sheets in turn list the products that 
resulted from the project. Included on the project sheets are the: 
project abstract; product outcome (with products classified as research 
products or technological products); completion dates; name of princi- 
pal investigator; primary and secondary audiences (l.e., users of 
product) ; and a summary of product evaluation. Each project sheet is 
placed within the catalog under one of thirteen major categories, and 
cross-references to other appropriate categories are also shown. It is 
anticipated that the catalog will be updated regularly. 

3. Information on educational products is also available from the 
Educational Products Information Exchange (EPIE) Institute. The purpose 
of EPIE is to provide its members with information based on studies of 
availability, use, and effectiveness of educational materials, equip*-' 
ment, and systems. Services are designed to facilitate the making of 
informed, rational evaluations of products by the consumers. EPIE was 
established in 1966 to provide up-to-date information on materials , 
equipment, and systems being produced for instruction, and also to 
provide guidelines for educators to use in selecting these products. 

EPIE uses the magazine Educational Product Report ( EPR ) as an instru- 
ment for dissemination of information to subscribers. Each issue 
centers on one type of product; "hardware" such as projectors and tape 



recorders; "software" such as books and films; or possibly a technique, 
such as team teachings Elach copy of EPR has a matrix summarizing in~ 
formation on a large number of products relating to the theme of the 
issue. This information is obtained from the producer and is fairly 
concise. EPR also includes articles which discuss the uses or signi- 
ficance of the type of product being reported, and generally an 
article either providing guidelines for selection of a product or pre- 
senting a model for evaluation of the effectiveness of the product. 

In a few cases EPR contains user reports or evaluation studies, but 
in general there is no attempt to evaluate the effectiveness of a 
product, to describe any one version of the product in detail, or to 
judge the merits of the different versions of the product. 

4. The PARaDE (^roducts/Accomplishments from JR ^ in Education) 

System is currently under development by the Office of Education 
National Center for Educational Research and Development to identify, 
catalog, and describe product outcomes of the educational laboratories 
and R&D centers. One objective of the system is to create a new data 
bank or information collection which can be used to: respond to 

management inquiries; interest agencies in laboratory and center pro- 
ducts; reply to product information requests; and fill a major portion 
of the product information needs growing out of a new laboratory and 
centers evaluation system. A second objective is to reduce the amount 
of time and effort spent by each laboratory and center in responding 
to product information requests. A third objective is to provide for 
the possibility of analyzing and reducing the raw data into a series 
of publications showing what the laboratories and centers have 
accomplished. 

Information is reported for the PARaDE system on two forms, one 
for developmental products and one for knowledge products (i.e., 
products contributing new information to our theoretical and technical 
knowledge base) . Information collected about developmental products 
includes: a description of the educational problem the product is 

designed to solve; the general strategy selected for the solution of 
the problem; release date; level of development; a description of the 
product; product users; product outcomes (i.e., expected changes 
resulting from product use); potential educational consequences; a 
list of product elements; start-up costs; operating costs; and likely 
market. Over 1,000 product reports have been received by the American 
Institutes for Research, which is collecting the information for NCERD 
under contract. 

5. The present project, the Evaluation of the Impact of Educa- 
tional Research and Development Products, has identified major sources 
of product development. The project has further developed some inex- 
pensive mail questionnaire procedures that can serve to obtain relevant 
product impact information. The actual questionnaire used in the pro- 
ject can be modified based upon the experience obtained in conducting 
the case studies and quantifying information for data analyses. By 
using such a questionnaire, a large percentage of product developing 
organizations can be inexpensively polled. This could be done on a 
regular basis to indicate changes in evidence of impact or other new 



data on previously reported products^ and to collect infornation on 
products developed since the last polling. 

6. An additional source of information on impact of products in 
use could be developed as part of current or future national reporting 
systems, such as the OE National Center for Educational Statistics 
Consolidated Program Information Report. Data regarding the use of 
educational products, particularly relatively new products, could 
easily be Included in a standard way in such reports. This wouil.d 
give an indication of the extent of use of products in all public 
schools rather than simply a sample. 



RECOMMENDATIONS FOR FURTHER RESEARCH 

From a study of this kind, many possibilities for further research 
can be suggested. Five possibilities, described below, are: an educa- 

tional analogue of Project TRACE; a study of factors related to adop- 
tion by users; adaption of the Product Development Reports for other 
audiences; the determination of factors predictive of later product 
impact; and the development of a system for monitoring product develop- 
ment. 



An Educational Analogue of Project TRACE 

Project TRACE, funded by the National Science Foundation, docu- 
mented the historical sequence of the key scientific events which led 
toward five major innovations. The, purpose was to study the contribu- 
tions of various types of R&D activity to innovations of importance. 

The study has had considerable impact. No analogue in education or 
the behavioral sciences exists. A systematic study of the role of 
research in an overall process which eventually leads to educational 
innovations would seem to be a highly worthwhile contribution. It 
could shed light on the role of the various mechanisms, institutions, 
funding activities, and types of research and development activity 
that lead to highly successful products. While there is some simi- 
larity to the current study, there are important differences. The 
current study focused on a number of specific products, tracing their 
particular development history at a relatively molecular level. It 
did not, as TRACE did, for example, attempt to delineate the scientific 
background of birth control pills, but rather the development of a 
particular X-company birth control pill. Nor is the current study 
designed to examine broadly for perhaps a 20 or 30 year preceding 
period those scientific and educational events leading to the develop- 
ment of a generic innovation. An educational Project TRACE, with 
certain modifications and perhaps some improvements based upon the 
examination of the completed TRACE project, could be of utmost impor- 
tance for R&D policy. 



Study of Factors Related to Adoption by Users 

A second extension of the present project would be an exploration 
of those factors considered important in the diffusion of a product 
by its development /diffusion agency, and those factors considered 
crucial by the potential target adopters or potential users. While 
there is a considerable amount of literature on characteristics 
related to diffusion, there is much less empirical data than hypotheses. 
Through actual case studies, the processes by which a potential user 
decides to adopt or not adopt a given product can be documented. The 
similarities and discrepancies between the strategies of the producer/ 
disseminator and the needs of the users could then be identified. 
Patterns for facilitating adoption processes could then be suggested, 
based on information about producer’s dissemination efforts and user s 
adoption processes. In addition, information about users' adoption 
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processes could be related to current Office of Education programs and 
policies, and recommendations prepared for consideration by Office of 
Education decision makers* 

The need for studies of this kind has been noted by Havelock 
(1971). Havelock's work is essentially an interpretive study. 

Although a large bibliography cites empirical studies, there is a 
conspicuous absence of empirical observations or tests of hypotheses 
in Havelock's volume. Of those empirical studies identified by 
Havelock, over 36% were concerned with "to whom" information was sent. 
For example, "who uses the county extension agent." ^ Fifteen percent 
of such studies were concerned with describing the "who" of the 
initiator or the originator of the message. In no case, in either 
Havelock's work or the references quoted, does there appear to be 
empirical comparisons between disseminator strategy and adopter per- 
ception. 

In fact, this kind of study is specifically recommended by 
Havelock. In the introduction the following statement is made: 

We were disappointed to find so few case studies . 

Of the thousands of dissemination and utilization 
events that take place each year it is unsettling 
to find so few documented in such a way that others 
could learn from them. This deficiency in the 
literature was one of the factors that thwarted 
our efforts to code, analyze, and compare utiliza- 
tion processes across studies and fields. Each 
investigator, in effect, has his own special 
interest or his own special point to make, and 
few appear to be motivated simply to report what 
happened in specific utilization events (p. 1-17). 

The first recommendation which Havelock derived from his over- 
view is: 



First of all although we noted there were many 
quantitative research studies, there was a paucity 
of case materials . We need more case studies 
which carefully document and report dissemination 
and utilization events. Such events, of course, 
come in all shapes and si^es, but we would include 
here training projects, development projects, and 
installation of new roles, and the development of 
new organization forms as all being activities 
requiring careful case reporting and documentation 

(p. 11-12). 

In addition, one of the ten research priorities identified by 
Havelock, and described as an area in which we know practically 
nothing, has to do ^th comparative evaluation of different knowledge 
utilization strategies. He notes that: 



. . . there is a dearth of well-documented and com- 

parable case study material on D&U, and we have urged 
that there be more studies along such lines provided 
they can be collected, compared, and summated in some 
way. A related need is for the systematic documenta- 
tion of specific strategies of utilization as employed 
by different change agents. Such studies should 
Include data on characteristics of senders and re- 
ceivers, messages, and media, a delineation of stages 
followed, implicit or explicit models of D&U employed, 
and results, assumed, observed, or measured (p. 11-36). 

Thus, the research proposed above complements and implements 
Havelock's work, and is consonant with one of his major recommenda 
tions for future research. 



Adaptation of the Present Product Development 
Reports for Other Audiences 

Consideration should be given to possible revisions and adapta- 
tions of some of the Product Development Reports. Although the case 
studies emphasized key events in the development of the products, some 
of the products are highly popular and of considerable interest to 
most people. Many of the products may be appropriately viewed as 
"trail blazers," or indicators of general movements in education. It 
may not, however, be anticipated that the Product Development Reports 
would have diffusion among teachers, superintendents, school boards, 
parents, etc. The reports could, however, be adapted for this 
audience in order to develop a more favorable public climate toward 
the investment of money in research, development, dissemination, and 
evaluation. Information could be provided regarding the instructional 
development process as currently and successfully followed and the 
resources demanded in successful development, diffusion, and evalua- 
tion. 



Determination of Factors Predictive of Later Product Impact 

It appears that many instructional products, if adequately docu- 
mented in the design and early developmental stages, could be evalu- 
ated for probable impact if appropriate analytical procedures were 
developed. This type of formative evaluation procedure could greatly 
increase the impact of research and development programs on education. 
In addition, decisionmakers should know the relationship between 
various factors in the research and development process and the 
expected degree of impact of educational products. Hence, a study 
should be undertaken to provide a more precise basis for the predic- 
tion of the impact of educational products than is currently avail- 
able. Using data available from the. current study and others, the 
logical and statistical relationships, between various aspects of the 
development and diffusion process can be studied as they relate to 
factors associated with successful impact. Well-known statistical 
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techniques such as multiple regression and factor analyses could be 
employed, and provisions made for cross-validation of the relation- 
ships derived in order to assure their generality to other products. 
Once the relationships were known, decisions regarding funding new 
product development, for example, could be facilitated by noting 
whether or not the proposals for such projects provided for the 
necessary elements related to later product Impact. 



Development of a System for Monitoring Product Development 

One method for increasing the effectiveness of the research and 
development program in the field of education would be the develop- 
ment of a system for the continuing evaluation of R&D products 
including data on both impact and estimates of probable iupact. The 
program would thus be under continuous audit with respect to the 
impact and anticipated impact of the various projects being supported 
in any given period. The development and implementation of such a 
continuous "early warning service" for product identification logi- 
cally stems from both the current study and the determination of 
factors predictive of later product Impact, as outlined above. 

Reviews of the Far West Laboratory ALERT System, the Information 
Office for the regional laboratories and R&D centers, and the work 
by EPIE suggest a need for both a more Inclusive data source and, 
particularly, for a continuous stream of evaluative information. 
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Table 1 

Focus of the 117 Selected Products 



Focus 


Freq. 


Percent 


Subject Matter 


103 


88.0 


Only 


100 


85.3 


And Perceptual-Motor 


2 


1.7 


And All Others 


1 


.9 


Organizational Efficiency 
Only 


3 


2.6 


0 


0.0 


And Classroom Climate 


1 


.9 


And Classroom Climate and 
Learning Procedures 


1 


.9 


And All Others 


1 


.9 


Classroom Climate or Operation 


7 


6.0 


Only 


0 


0.0 


And Organizational Efficiency 


1 


.9 


And Organizational Efficiency and 
Learning Procedures 


1 


.9 


And Learning Procedures 


4 


3.4 


And All Others 


1 


.9 


Learning Procedures or Methodology 


12 


10.3 


Only 


4 


3.4 


And Organizational Efficiency and 
Classroom Climate 


1 


.9 


And Classroom Climate 


4 


3.4 


And Perceptual-Motor 


2 


1.7 


And All Others 


1 


.9 


Perceptual-Motor Skills 


7 


6.0 


Only 


2 


1.7 


And Subject Matter 


2 


1.7 


And Learning Procedures 


2 


1.7 


And All Others 


1 


.9 



Note. — Percents do not add to 100 nor frequencies to 117 due to comblna 
tlons of focus.. 
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Note. — Percents do not add to 100 nor frequencies to 117 due to combinations of focus 
and grade level. 



Table 3 



Subiect Matter Content of the 117 Selected 


Products 




Subject Matter Content 


Freq. 


Percent 


Language 


A1 


35.0 


Only 


22 


18.8 


And Mathematics 


3 


2.6 


And Social Studies 


2 


1.7 


And Mathematics, Science, and Social Studies 


10 


8.5 


And All Others 


A 


3. A 


Mathematics 


30 


25.6 


Only 


9 


7.7 


And Language 


3 


2.6 


And Language, Science, and Social Studies 


10 


8.5 


And Science 


2 


1.7 


And Science and Social Studies 


2 


1.7 


And All Others 


A 


3. A 


Science 


A5 


38.5 


Only 


25 


21. A 


And Language, Mathematics, and Social Studies 


10 


8.5 


And Mathematics 


2 


1.7 


And Mathematics and Social Studies 


2 


1.7 


And Social Studies 


2 


1.7 


And All Others 


A 


3. A 


Social Studies 


A9 


A1.9 


Only 


29 


2A.8 


And Language 


2 


1.7 


And Language, Mathematics, and Science 


10 


8.5 


And Mathematics and Science 


2 


1.7 


And Science 


2 


1.7 


And All Others 


A 


3. A 


Vocational Education 


11 


9. A 


Only 


7 


6.0 


And All Others 


A 


3. A 


Note. — Percents do not add to 100 nor frequencies to 


117 due to 


comblna-* 



tlons of content. 
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Table 5 

Format of the 117 Selected Products 



Format 



Paper Products 
Only 

And Audio-Visual 

And Audio-Visual and Kits 

And Audio-Visual, Kits, and Computer 

And Kit 

And Computer 

By Paper Product: 

Textbooks Only 

Textbooks and Other Format (s) 
Workbooks Only 

Workbooks and Other Format (s) 
Manuals or Guides Only 
Manuals and Other Format (s) 

Tests Only 

Tests and Other Format (s) 



Audio-Visual 



Only 

And Paper Products 

And Paper Products and Kits 

And Paper Products, Kits, and Computer 

And Kit 

And Computer 

By Audio-Visual: 

Television Only 

Television and Other Format (s) 

Films (slides, strips, movie) Only 
Films and Other Format (s) 

Audio Equipment (records, tapes) Only 
Audio Equipment and Other Format (s) 



Kits and Equipment 
Only 

And Paper Products 

And Paper Products and Audio-Visual 

And Paper Products, Audio-Visual, and Computer 

And Audio-Visual 

CTable continued on next page) 



Freq . Percent 



99 84.6 



30 


25.6 


19 


16.2 


24 


20.5 


1 


.9 


19 


16.2 


6 


5.1 



10 


8.5 


65 


55.6 


0 


0.0 


24 


20.5 


8 


6.8 


68 


58.1 


0 


0.0 


22 


18.8 



47 


40.2 


1 


.9 


19 


16.2 


24 


20.5 


1 


.9 


1 


.9 


1 


.9 



0 


0.0 


2 


1.7 


0 


0.0 


40 


34.2 


1 


.9 


31 


26.5 



47 


40.2 


2 


1.7 


19 


16 • 2 


24 


20.5 


1 


.9 


1 


.9 



Table 5 (Continued) 



Format of the 117 Selected Products 



Format 


Freq. 


Percent 


Kits and Equipment (Continued) 






By Kits and Equipment: 






Kits Only 


0 


0.0 


Kits and Other Format (s) 


19 


16 . 2 


Displays (charts, maps, models) Only 


0 


0.0 


Displays and Other Format (s) 


15 


12 . 8 


Games Only , 


1 


.9 


Games and Other Format (s) 


19 


16 . 2 


Laboratory Equipment Only 


0 


0.0 


Laboratory Equipment and Other Format (s) 


15 


12.8 


Computer 


10 


8.5 


Only 


2 


1.7 


And Paper Products 


6 


5.1 


And Paper Products, Audio-Visual, and Kits 


1 


.9 


And Audio-Visual 


1 


.9 


Don’t Know 


11 


9.4 



Note. — Percents do not add to 100 nor frequencies to 117 due to combine 
tions of format. 
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Table 6 



Grade Level of the 117 Selected Products 



Grade 


Freq. 


Percent 


K 


50 


42.7 


1 


60 


51.2 


2 


60 


51.2 


3 


57 


48.7 


4 


53 


45.2 


5 


55 


47.0 


6 


58 


49.5 


7 


44 


37.6 


8 


50 


42.7 


9 


55 


47.0 


10 


49 


41.8 


11 


46 


39.3 


12 


44 


37.6 


Note. — Percents do not add to 100 nor frequencies 
product usually spans a number of grades. 


to 117 since a 



Table 7 

Grades Covered by the 117 


Selected 


Products 


Number of Grades Covered* 


Freq. 


Percent 


1 


10 


8.5 


2-3 


30 


25.6 


4-6 


35 


29.9 


7-9 


21 


17.9 


10-13 


21 


17.9 


*Re£ers to total number of grades, not 


levels 






90 

9 & 



Table 8 



Intended Percent of Target Population for the 117 Selected Products 



Target Population* 


Freq. 


Percent 


25% 


9 


7.7 


50% 


10 


8.5 


75% 


18 


15.4 


100% 


80 


68.4 



*Percent heading refers to percentage of that school population. 



Table 9 



Number of Schools in Which the 117 Selected Products Are in Use 



Number of Schools 


Freq. 


Percent 


5-100 


58 


49.5 


101-500 


34 


29.0 


501-1,000 


10 


8.6 


1,001-5,000 


10 


8.6 


5,001-35,000 


5 


4.3 
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Table 10 



Number of Students Using the 117 Selected Products 



Niunber of Students 


Freq. 


Percent 


100-1,000 


12 


10.3 


1,001-5,000 


20 


17.1 


5,001-10,000 


25 


21.3 


10,001-50,000 


29 


24.8 


50,001-100,000 


12 


10.3 


More than 100,000 


19 


16.2 



Table 11 



Year the 117 Selected Products Came Into Use 



Year Came Into Use 


Freq . 


Percent 


1965 


7 


6.0 


1966 


11 


9.4 


1967 


22 


18.8 


1968 


27 


23.1 


1969 


21 


17.9 


1970 


29 


24.8 
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Table 12 





Focus of the 117 Selected Products by Type of Developer 









Type of Developer 












Private 


Focus 




Government 


Profit-Making 


Non-Profit 


Subject Matter 


F 


48 


27 


28 


% 


41.0 


23.1 


23.9 


Organizational 


F 


0 


0 


3 


Efficiency 


% 


0 


0 


2.6 


Classroom Climate- 


F 


3 


0 


4 


Operation 


% 


2.6 


0 


3.4 


Learning Procedures- 


F 


6 


3 


3 


Methodology 


% 


5.1 


2.6 


2.6 


Perceptual-Motor 


F 


3 


3 


1 


Skills 


% 


2.6 


2.6 


.9 


Note. — Percents do not add 
tions of focus. 


to 100 nor 


frequencies to 117 


due to combina 



Content of 


Table 13 

the 117 Selected Products by Type of Developer 






Type of Developer 










Private 


Content 


Government 


Profit-Making 


Non-Profit 


Language 


F 15 


17 


9 




% 12.8 


14.5 


7 . 7 


Mathematics 


F 13 


7 


10 




% 11.1 


6.0 


,8.5 


Science 


F 21 


9 


15 




% 17.9 


7.7 


12.8 


Social Studies 


F 22 


10 


17 




% 18.8 


8.5 


14.5 


Vocational 


F 7 


1 


3 


Education 


% 6.0 


.9 


2.6 



Note. — Percents do not add to 100 nor frequencies to 117 due to combine- 
tlons of content* 
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Table 14 



Locations of the Developers of the 117 Selected Products 



Location of Developers 


Frequency 


Percent 


West 


35 


29.9 


North Central 


10 


8.5 


South Central 


1 


.9 


Northeas t 


63 


53.8 


Southeast 


8 


6.8 



Table 15 

Locations of the Developers of the 117 Selected Products 

By Type of Developer 



Type of Developer 



Location 
of Developer 




Government 


Profit-Making 


Private 

Non-Profit 


West 


F 


22 


7 


6 




% 


18.8 


6.0 


5.1 


North Central 


F 


8 


1 


1 




% 


6.8 


.9 


.9 


South Central 


F 


1 


0 


0 




% 


.9 


0 


0 


Northeast 


F 


19 


21 


23 




% 


16.2 


17.9 


19.7 


Southeast 


F 


5 


2 


1 




% 


4.3 


1.7 


.9 
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Table 16 



Distribution of 21 Exemplary Products Across 
Additional Selection Criteria and Constraints 



Focus^ 



Facilitating 

Lang. Math Science Soc. Studies Voc. Ed. Factors 
9 7 7 8 3 4 



Format 



The products were selected so as to represent: comprehensive 
formats with all the types of media; limited formats with 
only a textbook or manual and guide; kits; games; computer; 
television. 



Grade Level 



K-3 

6 

Extent of Use 

High 

17 

Gain High 

11 

Visibility 

M£h 

5 

Developer 

Government 

Agency 

9 



K-6 7-12 

7 3 



Moderate 

3 

Moderate 

8 



Moderate 

16 



Private Non-Profit 
Organization. 

7 



K-12 

5 



Low 

1 



Low 

2 



Low 

0 



Profit-Making 

Organization 

5 



^Frequencies total to more than 21 since some products have multiple 
foci. Data shown are not necessarily consistent with subsequent 
tables since product selection was based upon information from 
nominators as shown in Appendix E, 



Table 17 



Type of Developers of the 21 Exemplary Products 



Type of Developers 


Freq. 


Percent 


Regional Laboratory 


3 


1A.3 


Alone 


2 


9.5 


With R&D Center 


1 


A. 8 


R&D Center 


1 


A. 8 


Alone 


0 


0.0 


With Regional Laboratory 


1 


A. 8 


University 


8 


38.1 


Alone 


A 


19.0 


With Other Government 


2 


9.5 


With Private Non-Profit 


1 


A. 8 


With Profit-Making 


1 


A. 8 


Other Gi)vernment 


3 


1A.3 


Alone 


0 


0.0 


With University 


2 


9.5 


With Private Non-Profit 


1 


A. 8 


Private Non-Profit 


7 


33.3 


Alone 


A 


19.0 


With University 


1 


A. 8 


With Other Government 


1 


A. 8 


With Profit-Making 


1 


A. 8 


Profit-Making 


6 


28.6 


Alone 


A 


19.0 


With University 


1 


A. 8 


With Private Non-Profit 


1 


A. 8 



Note. — Percents do not add to 100 nor frequencies to 21 due to combina- 
tions of developers. 
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Table 18 



Type of Disseminator 


of the 21 Exemplary Products 




Type of Disseminator 


Freq. 


Percent 


Regional Laboratory 


3 


14.3 


Alone 


3 


14.3 


R&D Center 


0 


0.0 


University 


3 


14.3 


Alone 


0 


0.0 


With Profit-Making 


3 


14.3 


Other Government 


1 


4.8 


Alone 


1 


4.8 


Private Non-Profit 


5 


23.8 


Alone 


1 


4.8 


With Profit-Making 


4 


19.0 


Profit-Making 


15 


71.4 


Alone 


8 


38.1 


With University 


3 


14.3 


With Private Non-Profit 


4 


19.0 


No Disseminator 


1 


4.8 



Note. — Percents do not add to 100 nor frequencies to 21 due to combina' 
tlons of disseminators. 



O 
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Table 19 



Focus of the 21 Exemplary Products 


Focus 


Freq. 


Percent 


Subiect Matter 


18 


85.7 


Only 


11 


52.4 


And Learning Procedures 


7 


33.3 


Organizational Efficiency 


1 ' 


4.8 


Only 


1 


4.8 


Classroom Climate or Operation 


1 


4.8 


Only 


0 


0.0 


And Learning Procedures 


1 


4.8 


Learning Procedures or Methodology 


8 


38.1 


Only 


0 


0.0 


And Subject Matter 


7 


33.3 


And Classroom Climate 


1 


4.8 


Perceptual-Motor Skills 


1 


4.8 


Only 


1 


4.8 



Note. — Percents do not add to 100 nor frequencies to 21 due to combina 
tions of focus . 
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Table 20 





Grade Level of the 21 Exemplary Products 



Grade Level 


Freq. 


Percent 


Grade Level 


Freci^ 


Percent 


Kindergarten 


12 


57.1 


Grade 7 


9 


42.9 


Only 


2 


9.5 


Only 


0 


0.0 


K-2 


1 


4.8 


K-9 


1 


4.8 


K-3 


3 


14.3 


K-12 


3 


14.3 


K-6 


2 


9.5 


1-8 


2 


9.5 


K-9 


1 


4.8 


1-12 


2 


9.5 


K-12 


3 


14.3 


7-9 


1 


4.8 


Grade 1 


16 


76.2 


Grade 8 (Same as 


Grade 7 above) 


Only 

K-2 


0 

1 


0.0 

4.8 


Grade 9 


8 


38.1 








K-3 


3 


14.3 


Only 


0 


0.0 


K-6 


2 


9.5 


K-9 


1 


4.8 


K-9 


1 


4.8 


K-12 


3 


14.3 


K-12 


3 


14.3 


1-12 


2 


9.5 


1-6 


2 


9.5 


7-9 


1 


4.8 


1-8 


2 


9.5 


9-12 


1 


4.8 


1-12 


2 


9.5 


Grade 10 


6 


28.6 


Grade 2 (Same 


as Grade 1 above) 


Only 


0 


0.0 


Grade 3 


15 


71.4 


K-12 
1 o 


3 


14.3 

9.5 












Only 


0 


0.0 


9-12 


1 


4.8 


K-3 


3 


14.3 








K-6 


2 


9.5 


Grade 11 


7 


33.3 


K-9 

K-12 


1 

3 


4.8 

14.3 


Only 

K-12 


0 

3 


0.0 

14.3 


1-6 


2 


9.5 


1-12 


2 


9.5 


1-8 


2 


9.5 


9-12 


1 


4.8 


1-12 


2 


9.5 


11-12 


1 


4.8 


Grade 4 


12 


57.1 


Grade 12 (Same as 


Grade 11 


above) 








Only 


0 


0.0 








K-6 


2 


9.5 


• 






K-9 


1 


4.8 








K-12 


3 


14.3 








1-6 


2 


9.5 








1-8 


2 


9.5 








1-12 


2 


9.5 









Grade 5 (Same as Grade 4 above) 
Grade 6 (Same as Grade 4 above) 




Note. — Percents do not sum to 100 nor frequencies to 21 since products span a 
nvimber of grades. 
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Table 21 



Extent to Which Objectives for Student 
Performance Were Specified for the 21 Exemplary Products 







Percent 


Very General Objectives 


0 


0.0 


General Objectives 


3 


14.3 


Somewhat Specific Objectives 


4 


19.0 


Specific Obje^'tives 


6 


28.6 


Very Specific Objectives 


8 


38.1 
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Table 22 



Format of the 21 Exemplary Products as Used in the Schools 



Format 



Paper Products 
Only 

And Audio-Visual 

And Audio-Visual and Kits 

And Audio-Visual and Computer 

And Audio-Visual, Kits, and Computer 

And Kits 

And Kits and Computer 

By Paper Product: 

Textbooks Only 

Textbooks and Other Format (s) 
Workbooks Only 

Workbooks and Other Format (s) 
Manuals or Guides Only 
Manuals and Other Format (s) 

Tests Only 

Tests and Other Format (s) 



Audio-Visual 



Only 

And Paper Products 

And Paper Products and Kits 

And Paper Products and Computer 

And Paper Products, Kits, and Computer 

By Audio-Visual: 

Television Only 

Television and Other Format (s) 

Films (slides, strips, movie) Only 
Films and Other Format (s) 

Audio Equipment (records, tapes) Only 
Audio Equipment and Other Format(s) 



Kits 

Only 

And Paper Products 

And Paper Products and Audio-Visual 

And Paper Products, Audio-Visual, and Computer 

And Paper Products and Computer 

(Table continued on next page) 
101 



Freq. 


Percent 


21 


100.0 


3 


14.3 


2 


9.5 


7 


33.3 


1 


4.8 


1 


4.8 


6 


28.6 


1 


4.8 



0 


0.0 


8 


38.1 


0 


0.0 


15 


71.4 


2 


9.5 


19 


90.5 


0 


0.0 


15 


71.4 



11 


52.4 


0 


0.0 


2 


9.5 


7 


33.3 


1 


4.8 


1 


4.8 



0 


0.0 


1 


4.8 


0 


0.0 


8 


38.1 


0 


0.0 


10 


47.6 



15 


71.4 


0 


0.0 


6 


28.6 


7 


33.3 


1 


4.8 


1 


4.8 
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Table 22 (Continued) 

Format of the 21 Exemplary Products as Used in the Schools 



Format 


Freq. 


Percent 


Kits (Continued) 






By Kits: 






Displays (charts, maps, models) Only 


0 


0.0 


Displays and Other Format (s) 


12 


57.1 


Games Only 


0 


0.0 


Games and Other Format (s) 


7 


33.3 


Laboratory Equipment Only 


0 


0.0 


Laboratory Equipment and Other Format (s) 


5 


23.8 


Computer 


3 


14.3 


Only 


0 


0.0 


And Paper Products and Audio— Visual 


1 


4.8 


And Paper Products, Audio-Visual, and Kits 


1 


4.8 


And Paper Products and Kits 


1 


4.8 



Note. — Percents do not add to 100 nor frequencies to 21 due to combina- 
tions of format. 
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Table 23 



Format of the 21 Exemplary Products as Used in Training 


Personnel 


Format 


Freq. 


Percent 


Paper Products 


20 


95.2 


Only 


8 


38.1 


And Audio-Visual 


7 


33.3 


And Audio-Visual and Kits 


4 


19.0 


And Audio-Visual, Kits, and Computer 


1 


4.8 


By Paper Product : 


Textbooks Only 


0 


0.0 


Textbooks and Other Format (s) 


3 


14.3 


Workbooks Only 


0 


0.0 


Workbooks and Other Format (s) 


8 


38.1 


Manuals or Guides Only 


7 


33.3 


Manuals and Other Format (s) 


18 


85.7 


Tests Only 


0 


0.0 


Tests and Other Format (s) 


6 


28.6 


Audio-Visual 


12 


57.1 


Only 


0 


0.0 


And Paper Products 


7 


33.3 


And Paper Products and Kits 


4 


19.0 


And Paper Products, Kits, and Computer 


1 


4.8 


By Audio-Visual: 


Television Only 


0 


0.0 


Television and Other Format (s) 


1 


4.8 


Films (slides, strips, movie) Only 


0 


0.0 


Films and Other Format (s) 


11 


52.4 


Audio-Equipment (records, tapes) Only 


0 


0.0 


Audio Equipment and Other Format (s) 


7 


33.3 


Kits 


5 


23.8 


Only 


0 


0.0 


And Paper Products and Audio-Visual 


4 


19.0 


And Paper Products, Audio-Visual, and Computer 


1 


4.8 


By Kits: 


Displays (charts, maps, models) Only 


0 


0.0 


Displays and Other Format (s) 


3 


14.3 


Games Only 


0 


0.0 


Games and Other Format (s) 


2 


9.5 


Laboratory Equipment Only 


0 


0.0 


Laboratory Equipment and Other Format (s) 


1 


4.8 



(Table continued on next page) 
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Table 24 



Subject Matter Content of the 21 Exemplary Products 



Subject Matter Content 


Freq. 


Percent 


Language 


8 


38.1 


Only 


4 


19.0 


And Mathematics 


2 


9.5 


And Mathematics, Science, & Social Studies 


2 


9.5 


Mathematics 


6 


28.6 


Only 


2 


9.5 


And Language 


2 


9.5 


And Language, Science, & Social Studies 


2 


9.5 


Science 


5 


23.8 


Only 


3 


14.3 


And Language, Mathematics, & Social Studies 


2 


9.5 


Social Studies 


7 


33.3 


Only 


4 


19.0 


And Language, Mathematics, & Science 


2 


9.5 


And Vocational Education 


1 


4.8 


Vocational Education 


2 


9.5 


Only 


1 


4.8 


And Social Studies 


1 


4.8 


No Subject Matter Content 


2 


9.5 



Note. — Percents 
tlons of 



do not add to 100 nor frequencies to 21 due to comblna- 
subject matter content. 
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Table 25 



Cost of the 21 Exemplary Products to Users 



Cost/ Student/Year 


£re<^.. 


Percent 


$1 


3 


14.3 


$3 


2 


9.5 


$4 


2 


9.5 


$5 


3 


14.3 


$8 


2 


9.5 


$20 


1 


4.8 


$30 


1 


4.8 


$40 


1 


4.8 


$100 


1 


4.8 


$200 


1 


4.8 


No Information 


4 


19.0 
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Table 26 



Period of Student Use of the 21 Exemplary Products 



Period of Student Use 


Freq^. 


Percent 


Continuous Use (Hours Per Semester) 






20-29 


2 


9.5 


30-39 


3 


14.3 


40-49 


3 


14.3 


80-89 


1 


4.8 


90-99 


3 


14.3 


120-129 


1 


4.8 


130-139 


1 


4.8 


180-189 


1 


4.8 


Over 400 


1 


4.8 


Non— Continuous (Total Hours) 






1-9 


1 


4.8 


10-19 


1 


4.8 


Not Applicable 


3 


14.3 
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Table 27 



Number of Key Development Staff for the 21 Exemplary Products 



Number of Key Staff 


Freq. 


Percent 


1 


1 


4.8 


2 


2 


9.5 


3 


3 


14.3 


4 


3 


14.3 


5 


2 


9.5 


6 


5 


23.8 


7 


1 


4.8 


8 


1 


4.8 


12 


2 


9.5 


16 


1 


4.8 



Table 28 



Number of Key Development Staff for the 
21 Exemplary Products with Doctorates 



Number of Key Staff 
with Doctorates 


Freq. 


Percent 


0 


2 


9.5 


1 


3 


14.3 


2 


5 


23.8 


3 


3 


14.3 


4 


2 


9.5 


5 


1 


4.8 


6 


2 


9.5 


7 


1 


4.8 


12 


2 


9.5 
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Table 29 



Composition of Key Developmental Staff 
for the 21 Exemplary Products 



Type of Staff 


Freq . 


Percent 


Teachers 


2 


9.5 


Only 


0 


0.0 


And Faculty Members 


1 


4.8 


And Administrators and R&D Personnel 


1 


4.8 


College Faculty Members 


12 


57.1 


Only 


2 


9.5 


And Teachers 


1 


4.8 


And Administrators 


3 


14.3 


And R&D Personnel 


3 


14.3 


And Administrators and R&D Personnel 


3 


14.3 


Administrators 


11 


52.4 


Only 


0 


0.0 


And Teachers and R&D Personnel 


1 


4.8 


And Faculty Members 


3 


14.3 


And Faculty Members and R&D Personnel 


3 


14.3 


And R&D Personnel 


4 


19.0 


R&D Personnel 


15 


71.4 


Only 


4 


19.0 


And Teachers and Administrators 


1 


4.8 


And Faculty Members 


3 


14.3 


And Faculty Members and Administrators 


3 


14.3 


And Administrators 


4 


19.0 



Note. — Percents do not add to 100 nor frequencies to 21 due to combina- 
tions of type of staff. 
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Table 30 



How User Needs Were Determined Prior to the 
Development of the 21 Exemplary Products 



How Needs Were Determined 


Freq. 


Percent 


Educated Guess Based on Past Experiences 


21 


100.0 


Only 


2 


9.5 


And Asked Knowledgeable People 


9 


42.9 


And Obseirvatlon 


1 


4.8 


And Asked and Observation 


4 


19.0 


And Asked and Performance Measures 


1 


4.8 


And All Others 


4 


19.0 


Asked Knowledgeable People or Reviewed Literature 


18 


85.7 


Only 


0 


0.0 


And Educated Guess 


9 


42.9 


And Educated Guess and Obseirvatlon 


4 


19.0 


And Educated Guess and Performance Measures 


1 


4.8 


And All Others 


4 


19.0 


Observation of User Sample 


9 


42.9 


Only 


0 


0.0 


And Educated Guess 


1 


4.8 


And Educated Guess and Asked 


4 


19.0 


And 'All Ithers 


4 


19.0 


Performance Measures 


5 


23.8 


Only 


0 


0.0 


And Educated Guess and Asked 


1 


4.8 


And All Others 


4 


19.0 



Note. — Percents do not add to 100 nor frequencies to 21 due to combina- 
tions of how needs were determined. 
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Table 31 



Total Cost to Get the 21 Exemplary Products 
From Origin to User 



Dollars In Thousands 


Freq. 


Percent 


50 


1 


4.8 


90 


1 


4.8 


280 


1 


4.8 


360 


1 


4.8 


480 


1 


4.8 


1,000 


2 


9.5 


1,500 


1 


4.8 


2,000 


1 


4.8 


2,250 


1 


4.8 


3,000 


2 


9.5 


4,000 


2 


9.5 


5,000 


2 


9.5 


9,000 


1 


4.8 


14,000 


1 


4.8 


No Information 


3 


14.3 
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Table 32 



Procedures Used for First Cycle of Formative Evaluation 
of the 21 Exemplary Products 



Procedure Used 


Freq. 


Percent 


Development Staff Taught Using Product 


9 


42.9 


Only 


1 


4.8 


And Asked Teacher and Staff Observed 


1 


4.8 


And Staff Observed and Performance Measures 


1 


4.8 


And All Others 


6 


28.6 


Asked Classroom Teacher 


17 


81.0 


Only 


0 


0.0 


And Staff Taught and Staff Observed 


1 


4.8 


And Staff Observed 


4 


19.0 


And Staff Observed and Performance Measures 


5 


23.8 


And Perfomance Measures 


1 


4.8 


And All Others 


6 


28.6 


Development Staff Observed in Use 


19 


90.5 


Only 


1 


4.8 


And Staff Taught and Asked Teacher 


1 


4.8 


And Staff Taught and Performance Measures 


1 


4.8 


And Asked Teacher 


4 


19.0 


And Asked Teacher and Performance Measures 


5 


23.8 


And Performance Measures 


1 


4.8 


And All Others 


6 


28.6 


Obtained Performance Measures 


14 


66.7 


Only 


0 


0.0 


And Staff Taught and Staff Observed 


1 


4.8 


And Asked Teacher 


1 


4.8 


And Asked Teacher and Staff Observed 


5 


23.8 


And Staff Observed 


1 


4.8 


And All Others 


6 


28.6 



Note. — Percents do not add to 100 nor frequencies to 21 due to combina' 
tions of procedures. 
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Table 33 



Procedures Used for Second Cycle 
of Formative Evaluation of the 21 Exemplary Products 



Procedure Used Freq^ 

No Second Cvcle 6 


Percent of 

Products 
21 With 2nd 

Products Cycle 

28.6 


Development Staff Taught Using Product 


5 


23.8 


33.3 


Only 


0 


0.0 


0.0 


And Asked Teacher and Staff Observed 


1 


4.8 


6.7 


And All Others 


4 


19.0 


26.7 


Asked Classroom Teacher 


15 


71.4 


100.0 


Only 


0 


0.0 


0.0 


And Staff Taught and Staff Observed 


1 


4.8 


6.7 


And Staff Observed 


3 


14.3 


20.0 


And Staff Observed and Performance Measures 


6 


28.6 


40.0 


And Performance Measures 


1 


4.8 


6.7 


And All Others 


4 


19.0 


26.7 


Development Staff Observed in Use 


14 


66.7 


93.3 


Only 


0 


0.0 


0.0 


And Staff Taught and Asked Teacher 


1 


4.8 


6.7 


And Asked Teacher 


3 


14.3 


20.0 


And Asked Teacher and Performance Measures 


6 


28.6 


40.0 


And All Others 


4 


19.0 


26.7 


Obtained Performance Measures 


11 


52.4 


73.3 


Only 


0 


0.0 


0.0 


And Asked Teacher 


1 


4.8 


6.7 


And Asked Teacher and Staff Observed 


6 


28.6 


40.0 


And All Others 


4 


19.0 


26.7 


Note. — Percents do not add to 100 nor frequencies 


to 21 due to combinations 



of procedures. 
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Table 34 



Procedures Used for Third and Subsequent Cycles of 
Formative Evaluation of the 21 Exemplary Products 



Procedures Used 


Freq.« 


Percent 


of 








Products; 






21 


With 






Products 


Cycle 


Third Cycle 








No Third Cycle 


10 


47.6 




Development Staff Taught Using Product 


3 


14.3 


27.3 


Only 


0 


0.0 


0.0 


And Asked Teacher and Staff Observed 


1 


4.8 


9.1 


And All Others 


2 


9.5 


18.2 


Asked Classroom Teacher 


11 


52.4 


100.0 


Only 


0 


0.0 


0.0 


And Staff Taught and Staff Observed 


1 


4.8 


9.1 


And Staff Observed 


2 


9.5 


18.2 


And Staff Observed and Performance Measures 


5 


23.8 


45.5 


And Performance Measures 


1 


4.8 


9.1 


And All Others 


2 


9.5 


18.2 


Development Staff Observed in Use 


10 


47.6 


90.9 


Only 


0 


0.0 


0.0 


And Staff Taught and Asked Teacher 


1 


4.8 


9.1 


And Asked Teacher 


2 


9.5 


18.2 


And Asked Teacher and Performance Measures 


5 


23.8 


45.5 


And All Others 


2 


9.5 


18.2 


Obtained Performance Measures 


8 


38.1 


72.7 


Only 


0 


0.0 


0.0 


And Asked Teacher 


1 


4.8 


9.1 


And Asked Teacher and Staff Observed 


5 


23.8 


45.5 


And All Others 


2 


9.5 


18.2 



Subsequent Cycles 








No Subsequent Cycle 


16 


76.2 


— 


Development Staff Taught Using Product 


2 


9.5 


40.0 


Only 


0 


0.0 


0.0 


And All Others 


2 


9.5 


40.0 



(Table continued on next page) 
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Table 34 (Continued) 



Procedures Used for Third and Subsequent Cycles of 
Formative Evaluation of the 21 Exemplary Products 



Procedures Used 


Freq. 


Percent 


of 








Products 






21 


With 






Products 


Cycle 


Asked Classroom Teacher 


5 


23.8 


100.0 


Only '• 


0 


0.0 


0.0 


And Staff Observed 


1 


4.8 


20.0 


And Staff Observed and Performance Measures 


1 


4 . 8 


20.0 


And Performance Measures 


1 


4.8 


20.0 


And All Others 


2 


9.5 


40.0 


Development Staff Observed in Use 


4 


19.0 


80.0 


Only 


0 


0.0 


0.0 


And Asked Teacher 


1 


4.8 


20.0 


And Asked Teacher and Performance Measures 


1 


4.8 


20.0 


And All Others 


2 


9.5 


40.0 


Obtained Performance Measures 


4 


19.0 


80.0 


Only 


0 


0.0 


0.0 


And Asked Teacher 


1 


4.8 


20.0 


And Asked Teacher a id Staff Observed 


1 


4.8 


20.0 


And All Others 


2 


9.5 


40.0 



Note. — Percents do not add to 100 nor frequencies to 21 due to combinations 
of procedures. 
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Table 36 



Large and Small Scale Field Tests for 
Each of the 21 Exemplary Products 



No. of 
Field Tests 


Small 

_F 


Scale 

_% 


Large 

F 


Scale 

_% 


All 

Field Tests 
^ -1 


None 


12 


57.1 


8 


38.1 


5 


23.8 


1 


6 


28.6 


6 


28.6 


6 


28.6 


2 


0 


0.0 


3 


14.3 


4 


19.0 


3 


1 


4.8 


2 


9.5 


2 


9.5 


4 


0 


0.0 


1 


4.8 


0 


0.0 


5 


1 


4.8 


0 


0.0 


2 


9.5 


6 


0 


0.0 


0 


0.0 


1 


4.8 


No Information 


1 


4.8 


1 


4.8 


1 


4.8 



Note. — Data are based upon information available to the project staff as 
reported in the Product Development Reports. 
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Table 37 



Funding Source for the Major Field Test 
Each of the 21 Exemplary Products 


of 




Funding Source 


Freq., 


Percent 


Regional Laboratory 


4 


19.0 


Alone 


3 


14.3 


And Other Government 


1 


4.8 


R&D Center 


0 


0.0 


University 


2 


9.5 


Alone 


0 


0.0 


And Other Government 


2 


9.5 


Other Government 


10 


47.6 


Alone 


2 


9.5 


And Regional Laboratory 


1 


4.8 


And University 


2 


9.5 


And Private Non-Profit 


2 


9.5 


And Private Non-Profit and Profit-Making 


1 


4.8 


And Profit-Making 


2 


9.5 


Private Non-Profit 


3 


14.3 


Alone 


0 


0.0 


And Other Government 


2 


9.5 


And Other Government and Profit-Making 


1 


4.8 


Profit-Making 


4 


19.0 


Alone 


1 


4.8 


And Other Government 


2 


9.5 


And Other Government and Private Non-Profit 


1 


4.8 


Not Applicable (No Field Tests) 


5 


23.8 


No Information 


2 


9.5 




Note. — Percents do not add to 100 nor frequencies to 21 


due to 


combinations 



of funding source. 
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Table 38 



Geographical Extent of Field Tests of the 21 Exemplary Products 



No. of States 


Freq. 


Percent 


1 State 


4 


19.0 ' 


2 States 


1 


4.8 


3 States 


3 


14.3 


4 States 


3 


14.3 


7 States 


1 


4.8 


11 States 


1 


4.8 


Not Applicable (no field tests) 


5 


23.8 


No Information 


3 


14.3 
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Table 39 (Continued) 
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Mixed Cognitive & Mixed Affective 

Gains 1 ^-8 0 0.0 0 0.0 1 4.8 

(Table continued on next page) 
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Table 41 



Extent to Which the 21 Exemplary Products Were 
Modified as a Result of Major Field Tests 



Extent of Modification 




Percent 



Not at All 
Only a Little 
Somewhat 
Much 

Very Much 
No Field Tests 
No Information 



1 

7 

4 
2 
0 

5 
2 



4.8 

33.3 

19.0 

9.5 



0.0 



23.8 



9.5 
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Table 42 



Type of Marketer or Distributor 


of the 21 Exemplary 


Products 


Type of Marketer or Distributor 


Freq . 


Percent 


Regional Laboratory 


2 


9.5 


Alone 


1 


4.8 


And Profit-Making 


1 


4.8 


R&D Center 


0 


0.0 


University 


0 


0.0 


Other Government 


1 


4.8 


Alone 


1 


4.8 


Private Non-Profit 


2 


9.5 


Alone 


0 


0.0 


And Profit-Making 


2 


9.5 


Profit-Making 


18 


85.7 


Alone 


15 


71.4 


And Regional Laboratory 


1 


4.8 


And Private Non-Profit 


2 


9.5 


No Marketer or Distributor 


1 


4.8 




Note. — Percents do not add to 100 nor 


frequencies to 21 due to combina— 



tlons of marketers or distributors. 
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Table 44 



Extra Staff Requirements 


for Use of the 21 Exemplary Products 


Extra Staff Requirements 


Freq . 


Percent 


Supervision 


4 


• 19.0 


Only 


1 


4.8 


And Paraprofessional 


1 


4.8 


And Consultants 


1 


4.8 


And All Others 


1 


4.8 


Paraprofessional 


5 


23.8 


Only 


3 


14.3 


And Supervision 


1 


4.8 


And All Others 


1 


4.8 


Additional Teachers 


2 


9.5 


Only 


1 


4.8 


And All Others 


1 


4.8 


Consultants 


4 


19.0 


Only 


2 


9.5 


And Supervision 


1 


4.8 


And All Others 


1 


4.8 


None 


11 


52.4 


Note. — Percents do not add to 


100 nor frequencies to 21 


due to combine- 



i tions of requirements. 
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Table 45 



Methods of Obtaining Feedback From Users of the 21 Exemplary Products 



Method Freq . Percent 



Informal Collection of Information 


19 


90.5 


Only 


15 


71.4 


And Systematic Sampling 


3 


14.3 


And All Others 


1 


4.8 


Systematic Sampling 


5 


23.8 


Only 


1 


4.8 


And Informal Collection 


3 


14.3 


And All Others 


1 


4.8 


Survey of All Users 


1 


4.8 


Only 


0 


0.0 


And All Others 


1 


4.8 


None 


1 


4.8 



Note. — Percents do not add to 100 nor frequencies to 21 due to combina' 
tions of methods. 
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APPENDIX A 



SUMMARY OF THE STUDY'S OBJECTIVES AND APPROACH; 
INSTRUCTIONS FOR THE PRODUCT RATING FORM 





( 



ERIC 



A-l 



ISS 



AMERICAN INSTITUTES FOR RESEARCH 



Center for Research and Evaluation 

in the Applications of Technology in Education 



The American Institutes for Research, with support from the U. S. Office of 
Education, is examining the process of development of "successfixl educational 
products." A number of products which have demonstrated a successful impact 
in schools will be identified. A selected set of those products identified 
will then be extensively reviewed. The review will focus upon the development 
and diffusion history of the product. A major objective of these reviews is to 
obtain empirical data regarding those development and dissemination processes 
which affect product impact. 

To accomplish this, we need to identify a number of educational products that 
have had a productive impact. We are requesting your help in identifying some 
of these products. Would you nominate some "successful educational products"? 

These products may be tangible, such as a text, or intangible, such as a teaching 
technique. They may be products that have been developed through federal funds, 
non-profit foundations and organizations, or through private enterprise. Their 
development may have involved your organization, or they may be products familiar 
to you but developed by others . 

The concern is with products that have "made it" during the last few years and 
are in use in grades K through 12. Products that have a traceable history of 
development and show some indication of contributing, directly or indirectly, 
to measurable performance gains are of major interest. 

Enclosed is a complete list of the criteria with explanatory notes for use in 
identifying products. On separate sheets, you will find rating forms for pro- 
ducts. Use one rating form for each product identified. Identify as many pro- 
ducts meeting the criteria as your time and familiarity with the products permit. 
An addressed and stamped envelope is enclosed for the return of rating forms. 

We appreciate your cooperation. If you wish, we will keep you informed of the 
progress and accomplishments of this project. If you have any questions regard- 
ing the project of these procedures, please phone Carolyn Morrow, Dan Kratochvil, 
or me collect at (1^15) 328-3550, extension 97. 



Sincerely, 



Enclosures 
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Jack J. Crawford, Associate Director 
Evaluations and Research Program 



♦ > ••• 

0 I -RO CREATE □ po 80^]^ 1 , 3 . PAL.O ALTO. CALIFORNIA 94302 (LOCATION: 1791 ARASTRADCRO ROAD) □ TEL. (415) 328*3550 



INSTRUCTIONS FOR THE PRODUCT RATING FORM: 
CRITERIA FOR SELECTING EDUCATIONAL PRODUCTS WITH "IMPACT” 



Read the following criteria and their explanatory notes, and then use them as a 
guide in completing the product rating forms. The criteria are in italics. 

I. Definitional Criteria 

Definitional criteria are those which will be applied initially to distinguish 
educational products from non-products for the purpose of study delimitation. 
These criteria will insure that products are construed as having behavioral 
as well as physical attributes and as having evolved from empirical research 
and developmental procedures. While this is an OE supported project, proprietary 
products as well as products of OE supported projects should be considered. A 
product may be either tangible or intangible; that is, it may be a physical 
entity or it may be a dynamic process having clearly delineated components 
consistent across settings. 

t 

A.l. The product should have explicit and well-defined goals and 
objectives. Or^ 

2. The product should have implicit objectives of major importance. 

Written formulation of goals and/or objectives may appear as 
statements of purpose or outcome. Goals should relate clearly 
to some broad conceptual scheme (such as curriculum) or a 
methodology (such as a teaching-learning technique) . Objectives 
should indicate the target audience, conditions, and anticipated 
indicators of success. 



B. l. The product should have procedures and guidelines for its 

implementation and use. And^ ■> 

2. The product should specify the texts ^ equipment, or techniques 
which are to be employed. 

The product may be a discrete "unit” or it may be a complex of 
things that are seen as integral and coherent. 

C. l. Systematic data regarding the development and use of the product 

in the field should be available. 

It should be possible to obtain some type of existing systematic 
data regarding the product’s development and use in the field so 
that the achievement of goals or objectives could be assessed. A 
high degree of behavioral specificity would be desirable. 



II. Impact Criteria 

Impact criteria are those which will be applied to determine the importance 
of the product as a contribution to educational practice. This importance 
will be demonstrated along two dimensions: the scope or magnitude, and the 
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effect or intensity of intended outcomes. These criteria will insure that 
comprehensive products having impact on a large scale, over time, will be 
Included and that these products will have demonstrable results on the 
target audience . 

A. Scope 

1. a. The product must be implemented in one of the following grades: 

K-1-2-3-4-5-6-7-8-9-10-11-22 

b. The product must be implemented in one of the following areas: 

Language^ mathematics 3 social studiesy sciencesy or 
vocational education 

The product must be implemented at some point within the range 
of grades and subject matter areas specified above. The pro- 
duct cannot be college level, conducted in private or by 
correspondence, or preschool when not in association with the 
school program. Either it involves content in one of the 
mentioned disciplines or it is a process applicable to these 
disciplines. 

2. The product must have presently available a full and complete 
written description in sufficiently definitive fom that it 
can he installed as described. 

A "full and complete description" would include information of 
the type categorized under the heading Definitional Criteria. 

3. The product should be in use in at least five schools having no 
direct connection with the original developers. 

The intent of this criteria is to exclude those products that 
work due primarily to the efforts of one person. 

4. The target population at which the product is aimed should he 
a substantial one. 

"Substantial" should be defined as at least a fourth of the 
total population in U. S. schools at the intended age or grade 
level . 

5. The product should have been brought to the attention of the 
educational profession through professional journalsy publica- 
tionSy conventionsy or other broad dissemination procedures 
such as marketing technicfues. 

Reference to the product should exist in the literature in more 
than one .journal at a minimum, and its existence should be "known" 
by some reasohable proportion of professionals working in the 
discipline. 
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The px’oduct should have oome ‘into use in the sohooZs duving^ 
the last five nears. 

A product could be "emerging" at the present time or could 
have been in existence at any time during the last five years. 

B. Effects (The product should meet at least one of the following two 
conditions. ) 

1. The pToduot should hate produced results luhioh suggest a 
measurable gain toward accepted educational goals relatite ^ 

to a specified baseline ^ and these results should be sustained 

over time. 

Measurable gain should be indicated by the results of a controlled 
experiment in which the norm or starting point foi the target 
audience was given and in which comparative data over time was 

obtained . 

2. If the product impact is not directly measurable in terms of 
educational goalsy the product should hate contributed toward 
accepted educational goals in at least one of the following 
af'eas : 



a. School organizational efficiency 

b. Classroom climate or operation 

Q, Learning procedures or methodology 
d. Improved perceptual-motor shills 

The "areas" listed will, presumably, have a bearing on student 
development by modifying environment, human interactions, or 
skills when the product is used. "Learning procedures or 
methodology" includes students learning about themselves or 
the world of work. 
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PRODUCT RATING FORM 



Respondent Name Phone 

Respondent Organization 

Name of Product 

Product Developed By 

Directions . Circle the appropriate answer (Y = Yes, N = No, DK = Don't Know) or fill 
in the blanks. 



I. Definitional Criteria 

A. l. Has explicit objectives? 

2. Has implicit objectives of major Importance? 

B. l. Has procedures and guidelines for use? 

2. Specifies materials and techniques? 

C. l. Data available regarding development and use? 



Y N DK 

Y N DK 

Y N DK 

Y N DK 

Y N DK 



II. Impact Criteria 
A. Scope 

1. a. Circle grades within which the product was implemented: 

K12 34567 89 10 11 12 

b. Circle areas within which the product was implemented: 
Language Mathematics Science Social Studies 
Vocational Education 

2. Description available so product can be installed 

as described? ^ ' 

3. Estimate number of schools and number of students 

using the product: Schools Students 



4. Estimate the percentage of school population, at the 

intended grade level (s), for which the product is 
designed: (Circle) 

5 % 10% 25% 50% 75% 100% 

5. Estimate amount of dissemination of the product by 
writing "high," "medium," or "low": 

6. Indicate the year when the product came into classroom 
use: 
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Effects 

1. Results indicate measurable gain? 

2. Product contributed toward: 

School organizational efficiency? 
Classroom climate or operation? 
Learning procedures or methodology? 
Perceptual-motor skills? 
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Y N DK 

Y N DK 

Y N DK 

Y N DK 

Y N DK 
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PRODUCT DESCRIPTION FORM 



I . Product 

A. Name: 

B. Acronym: 

II. Developer 

A. Name: 

B. Address (organization address if relevant): 



C . Phone : 

III. Disseminator 

A. Name: 

B. Address: 

C . Phone : 

IV. Users 

A. Summary of ^Schools and ^Students: 



B. Sites where operating successfully: 



V. Descriptive Summary 

A. Focus (use) of product (subject matter, organization, classroom climate ^or 
procedures, learning procedures or methodology, perceptual-motor skills). 

If subject matter, indicate — basic course more than 1 year, basic course 1 
year, basic course less than 1 year, supplementary units or materials, 
organizational patterns only, manual/guides only. If not subject matter, 
indicate people in addition to students — parents /community group, administra- 
tors/supervisors, teachers and paraprofessionals— and school related things-- 
room arrangement, scheduling — that are dealt with. 
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B. Content (language, math, science, social studies, vocational education), 



C. Grade Level (K, 1, 2, 3, U, 5, 6, 7, 8, 9, 10, H, 12). 



D. Target Population (% of school population at the intended levels) 



E. Oh.lectivcs 



(objectives of product including sViort and long range goals) 



f 

•'i 

<^1 



F. Anoroach to Subjec t Area (c.g., non-graded, individualized, traditional, 
prograrjned Tnst^ction, computer assisted instruction, parent-community 
involvement, student self direction, discovery/inquiry. } 



G. Format— media (textbooks— teacher , student; ’^°-^tooks— lab,^supplem^^_^^> , 
worksheets; manuals/guidcs; .television series; films-movies, fitastrips, 
filmloops, slides; audio equipaent-records , tapes, cassettes, 
charts, graphs, models, maps, globes, atlas, flash cards; kits, tests, games, 
lab equipment; computer and related equipment). 



H. Number of instructional hours and recommended period of use 
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VI. 



VII. 



VIII. 



IX. 



o 

ERIC 



I. staff training requirements: 



Summary of Procedures and Guidelines : 



Summary of Development and Diffusion Schedule: 



Measured Cocnitive Effects (subjects, behaviors/outcomes measured, measT^es employed, 
control groups, immediate"results , sustained gains, who did research and when): 



Measured Affective Effects (subjects,' behaviors /out comes measiired, meas\ires employed, 
control groups, immediate results, sustained gains, who did research and when): 



■ 
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X. 



contribution Toward Accepted Educa tional Goa ls throuch:^ ’ 

classroon ciimatrana operation, learning prlJc eduros and methodology, Pf 
motor skills. (Subjects, behaviors/out comes measured, measures employed, control 
groups, immediate results, sustained gains, who did research and when.; 
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APPENDIX D 



BRIEF NARRATIVE DESCRIPTIONS OF THE 117 SELECTED PRODUCTS 



America: A Modern History of the Ilnltod StatP-s wa;; developed and is 

distributed by D.C. Heath and Company. In use since i07 ■ and designed for 
75% of 11th grade social studies students, the product makes use of a text, 
teacher’s edition, records, and tests. The student is encouraged to review 
important historical decisions, analyze the conseauences of the decisions, 
and ccTf' to his <■ n conclusins recarding thcr 

The Amherst Project Materials , in use since 1^70. were developed by 
.Amherst College and distributed by Hampshire College. Designed for 757.. 
of grade 9 through 12 social studies students, the materials aim at lead- 
ing students to discover some of the concepts that historians use in develop- 
ing their explanations of trends and events in American b.istory. Texts, 
manuals, and audio— sual aids comprise the materials. Tn— service training 
workshops for teachers and administrators are sponsored by the project. 

The Anthropology Curriculum Project Materials were developed by the 
University of Georgia and have been in use since 1965. A social studies 
curriculum for all students in grades K-12, the product consists of sequen- 
tial units in anthropology which aid students and teachers in using 
anthropological concepts as a means of developing cross cultural perspec- 
tives on human behavior. Materials include texts, guides, background 
materials, and tests. 
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The Anthropology Curriculum Study Project Materials were developed 
by M'lcclr. Collier and the American Anthropological Association and have 
been in use since 1968. Designed for all social studies students in 
grades 7 through 12, materials include texts, workbooks, manuals, audio- 
visual aids, and displays. The purpose of the product is to help students 
understand anthropological concepts at the junior and senior high school 
levels. 

A Place to Live , developed and distributed by the National Audubon 
Society Educational Service, came into use in 1970. Designed for all 
inner-city science students in grades 4 through 6, the student workbook 
and teacher’s manual aim at giving children an understanding of ecology 
even though they live in an urban area. Activities include outdoor "Walks" 
and "try this" exercises as well as workbook exercises. 

The Arithmetic Proficiency Training Program was developed by Science 
Research Associates and has operated since 1970. Designed for all mathe- 
matics students in grades 1 through 8, it is a computer— administered 
test-and-practice program designed to provide pupils with highly individu- 
alized supplementary work in arithmetic computation. 

The Bank Street College of Education Model was developed by the Bar*" 
Street College of Education and has been in use since 1969. With a focut> 
on learning procedures and methodology and on classroom climate, the model 
is designed for 50% of grade K-3 pupils in all subject areas. The ultimate 
objective is to enable each child, in his initial years of schooling, to 
build a positive image of himself as a learner. The teacher training pro- 
gram includes orientation institutes (workshop, field trips, seminars, and 
laboratory observations), on-site visits by the Bank Street staff, on-site 
seminars, workshops, and conferences. Schools using the Bank Street Model 
require a teacher and two aides per classroom and one administrative staff 
member per ten classrooms . 

The Behavioral Analysis Model was developed by Don Bushell of the 
University of Kansas and has operated since 1968. Designed for all pre- 
school through 3rd grade children of low-income families, the model, 
consisting of guidelines and training methods, focuses on learning pro- 
cedures and methodology and on classroom climate. Its goal is to teach 
the social and academic skills needed for success in school and concen- 
trates on the skills of reading, mathematics, spelling, and handwriting. 
The principles of behaviorist psychology are applied. Programmed materials 
are recommended but not provided. 

BSCS Laboratory Blocks were developed by the Biological Sciences 
Curriculum Study and have been in use since 1968. Designed for 75% of 
grade 9 through 12 science students, the blocks provide six-week programs 
of concentrated investigation into a single area of biology, thus guiding 
the student to his own discoveries and conclusions. 
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The Center for Environmental Education Material;; were developed by 
the Minnesota Environmental Sciences Foundation and have been in use 
since 1967. Designed for all students at all levels of science and social 
studies, the materials include inquiry-oriented texts, kits, and labora- 
tory equipment designed to help students inquire about their environment. 
Several teacher training classes are also offered. 

Chemistry; Experiments and Principles v;as developed by the University 
of California at Berkeley in 1968 as a revision of an earlier chemistry 
program. Designed for 25% of 11th and 12th grade science students, the 
materials produced form a course based upon experiments where students' 
observations and measurements lead to the dc:velopment of unifying principles. 
A text, laboratory manual, teacher's guide, pamphlets, tests, and films make 
UT) the student's materials. A variety of teacher trainina aids have also 
been produced. 

Cluster Concert Program-s were developed by i).'. Donald Maley and 
Dr. Walter Mietus of the University of Maryland and have been in use since 
1967. Designed for 50% of 11th and 12th grade vocational education students, 
the curriculum outlines and instructional plans aim at preparing youth for 
entry-level capabilitv in a variety of related oo. iijiations rather than a 
specific occupation. Materials include manuals and tests. 

The Comprehensive School Mathematics Program was developed by the 
Central Midwestern Regional Educational Laboratorv an I has operated since 
3970. Dcsicned as complete mathematics curricula for students at all levels, 
the program uses audio-visual aids, games, kits, and programmed texts. The 
curricula attempt to provide each student with a program that is sound in 
content, enjoyable, and appropriate to his abllitnes and needs. 

The Computer Assisted Instruction Drill-and-Practice Program in 
Elementary Mathematics was developed by Stanford University and has been 
in use since 1967. Designed as supplementarv work for all mathematics 
students in grades 1 through 6, the materials provide computer-assisted, 
self-paced instruction in the teaching of basic mathematics skills. It 
requires preservice instruction by the developer. 

Computer Assisted Instruction: Elementary En g lish was developed by 

tlie Instructional Systems Division of Harcourt Brace Jovarovich, Inc., and 
has been in use since 1969. Designed for all language students in grades 
4 through 6, the program provides computer-assisted, self-paced instruc- 
tion in teaching basic English skills. 

Computer-Assisted Instruction in Language Arts was developed by 
Stanford University and has been in use since 1968. It is designed for 
all language students Ln kindergarten through grade 3 and provides supple- 
mentary drill in basic decoding skills to complement any initial reading 
series to improve pupils' reading skills. Eight- to twelve-minute daily 
sessions at the computer are recommended. 
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The Conceptually Oriented Program in Elementary Science was developed 
by New York University and has been in operation since 1969. It is de- 
signed for all science pupils in grades K-6. The manuals and workbooks 
that comprise the material are based on the conceptual schemes approach 
and are action-centered. Nearly all activities require exploration of a 
non-reading nature. The ultimate goal is to give pupils an understanding 
of the nature of matter in terms of a few basic conceptual schemes. The 
program is expected to form about 80% of an elementary science curriculum. 
In-service training for teachers is recommended for grades 3 through b. 

Coordinated Helps in Language Development was developed by the North- 
west RegiojrH Educational Laboratory and has been in use since 1970. 

Designed as an aid for 50% of kindergarten pupils, the material, consisting 
of a teacher's guide, focuses on learning procedures and methods to help 
teachers conduct learning activities which will increase pupils oral lan- 
guage skills. 

The Co rrelated Curriculum Project Materials, , developed by the New York 
City Board of Education and the Ford Foundation, have been ^se since 
1966 and are designed for 50% of grade 9 through 12 vocational education 
students, .liurricul^^ materials aim at helping the non-academic student 

prepare m by means of exploratory courses in business, health, 

and industrf iaiad ^preparation for a careers program in junior college. 

The Cr eative Learning Group Drug Education Program , developed by the 
Creative Learning Group of Cambridge, Massachusetts, has been in use since 
1970. Designed for all social studies students in grades 6 through 10, the 
program provides students with accurate information (regarding drug use and 
abuse) to help them in making important personal decisions. Texts, manuals, 
and audio-visual aids comprise the materials. Fifteen to twenty class ses- 
slons are recommended. 



Dangerous Parallel , developed by the Foreign Policy Association and 
distributed by Scott-Foresman Publishing Company, is a simulation (game; 
designed for all social studies students in grades 9 through 12. The 
simulation allows students to represent six fictional nations in decision- 
making during international crises. Its aim is to involve students in 
the learning process and prepare them for subsequent learnii^. 



Demonstr ation and Experimentation in Computer Tra ining and Use in 
S econdary Schools Materials , developed by Thomas E. Kurtz, Kiewit Compu- 
tation Center of Dartmouth College, have been in use since 1968. Designed 
for all students in language, mathematics, science, and social studies in 
grades 9 through 12, the materials aim at utilizing a computer to aid 
students in learning a wide variety of school subjects. Computers, texts, 
manuals, and workbooks comprise the materials used. 
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The Development of Economic Education Prograin , developed by the 
Joint Council on Economic Education, has been in ti<?e since 1969 and the 
materials are designed for all social studies students at all levels. 

The texts, guides, and filmstrips that comprise the program aim at giving 
students a basic background in economics. 

The Development of Higlier Level Thinking Abilities , developed by 
the Northwest Regional Educational Laboratory, has been in use since 1966. 
With a focus on learning procedures and methodology and on classroom cli- 
mate , the materials aim at all teachers of grades 2 through 12 and prepare 
them to use teaching strategies which increase the abilities of students 
to solve problems by categorizing facts, forming generalizations, and 
applying the generalisations to unknown situations. Teacher training 
materials include a m.anual , film, and audio— tapes. 



The Discovery Approach to American History was developed by the 
Carnegie Education Center, Carnegie-Mellon University, and has been in 
use since 1968. Designed for 75% of grade 8 through 11 social studies 
students, the materials encourage students to develop into independent 
thinkers and responsible citizens by means of inquiry techniques. 

Materials include texts, kits, audio-visual aids, manuals, and tests. 

The DISTAR System tms developed by SieefrI' 1 r.nr'clnann and Associates 
and is distributed by S.Jencc Research Associate.^, Inc. It has been in 
use since 1969. Designed for 50% of K—2 language and mathematics pupils, 
DISTAR materials are aimed particularly at helping culturally disadvantaged 
children improve their language and mathematics skills through the use of 
behavioral objectives, teacher training, material development, pupil moti- 
vation. regular instruction, attention to individual learning problems, 
and parental involvement . 

Drug Abuse Curriculum materials, developed by the Laredo, Texas, 
Independent School District, has been in operation since 1970. Designed 
for social studies students at all levels, the instructional materials 
provide a drug education curriculum with a sequential grade approach in 
order to enable students to make informed personal decisions regarding 
drug use. 

Drug Decision, developed by Lockheed Educational Systems and in use 
since 1969, is designed for all science and social studies students in 
grades 6 through 10. The complete drug education program, using programmed 
instruction, films, and a game, is designed to give students an awareness 
of the problems and effects of drug abuse. The curriculum requires four 
hours of in-service teacher training and requires one hour daily of stu- 
dent use for 21 davs. 

Drug Experi ence Program? Data for Decision Making , developed by 
Dr. David C. Lewis and distributed by CSCS, Inc. has been in use since 
1970. Designed for all social studies students in grades 6 through 12, 
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the texts, tapes, manuals, and tests provide accurate information to 
students to enable them to make personal decisions regarding drug use. 

The materials comprise a one-semester course of 35 hours. 

T^arth Science Curriculum Project Materials , developed by William D. 

Romey and the American Geological Institute, have been in use since 1968. 
Designed for all science students in grades 8 through 10, the materials 
provide an interdisciplinary treatment and investigative approach to the 
earth and environmental sciences in order to give students a comprehensive 
view of the planet earth and its environment. Materials include texts, 
manuals, films, and laboratory equipment. 

The Econon^^c Man Program was developed by the University of Chicago 
Industrial Relations Center and is distributed by Benefit Press. Designed 
for all social studies students in grades 6 through 8, the materials strive 
to teach basic economic principles through use of model situations and a 
simulation game. The course encompasses one semester and uses texts, 
manuals, and games. 

The Educational Development Center Follow-Through Model , developed 
by Educational Development Center, has been in use since 1968. Designed 
for all pupils in kindergarten through grade 3, the model operates as a 
consulting service aimed at providing student self —direction in an open 
classroom" environment. It focuses on learning procedures and methodology 
and on classroom climate. Staff training is required and is provided by 
EDC advisors . 

Educational Programming of Cultural Heritage was developed by the 
Berkeley Unified School District and has been in use since 1966. Designed 
for all social studies pupils in grades K through 6, the product is a multi- 
media environment for learning in the form of a demonstration chamber that 
makes it possible to surround the pupil with a simulated environment in 
order to give him an understanding of man's cultural heritage. Chamber 
materials Include slides, texts, films, records, artifacts, tapes, and 
displays. A portable classroom unit that can be installed locally is 
under development. 

The Educati onal Research Council Science Program , developed by ^C, 
has been in dperation since 1967 and is designed for all science students 
in grades K-12. A complete science program with an articulated curriculum 
provides for independent study, laboratory investigations, discussion groups, 
computer— assisted instruction, individualized pacing, audio— tutorial in- 
struction, simulations, and games in order to improve students' science 
education. Teacher and student guides, tests, and resource books comprise 
the instructional materials. 

The Elementary Science Study Materials were developed by Educational 
Development Center and have been in use since 1965. Designed for all science 
students in kindergarten through 8th grade, the program is a highly individual. 
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experimental one in which all pupils have access to materials for open- 
ended rather than teacher- or text-directed investigations # Materials 
include texts, films, manuals, displays, and laboratory equipment to be 
used three hours a week for five to seven weeks. 

Facilitating In quiry in the Classroom , developed by Northwest Regional 
Educational Laboratory, has been in use since 1966. Not content bound, the 
materials are used by teachers at all levels in a wide variety of subject 
areas, most often in science. Focused on learning procedures and method- 
ology, the materials are designed to prepare teachers to perform 27 behaviors 
which encourage pupils to inquire and become autonomous learners. All 
materials (leader's guide, exercises, worksheets, evaluation forms, and 
audio-tapes) and procedures for using them to conduct 42 hours of instruc- 
tion as an in-servico workshop or college course are available. 

The Field Social Studies Program was developed by Dr. Richard E. Gross 
and Dr. John U. Michaelis and is distributed by Field Educational Publica- 
tions, Inc. In use since 1970, the program is for all social studies 
students in grades K-12 and represents a conceptual approach to social 
studies through the use of texts. One hour daily use is recommended to 
enable pupils to develop social studies inquiry skills and concepts. 

The First Year Communication Skills Program , developed by the Southwest 
Regional Laboratory for Educational Research and Development, has been in 
use since 1967. Designed for all kindergarten pupils, the program provides 
a set of research-based instructional materials and procedures for teacher 
use in developing the reading competence of young children. Materials 
include texts, games, displays, and manuals. 

The Frostlg Program for Perceptual -Mo tor Development , developed by 
Dr. Marianne Frostig and distributed by Follett Educational Corporation, 
has been in use since 1969. Focused on perceptual-motor skills and learn- 
ing procedures and methodology for 75% of pupils in levels K-2, the pro- 
gram is designed to enhance the total development of young children by 
improving and/or developing good health, physical well-being, sensory- 
motor skills, self-awareness, awareness of time and space, and the ability 
to communicate, to interact with others, to perceive self in relation to 
environment, to solve problems, and to learn. Materials Include audio- 
visual aids, guides, books, displays, and various materials for physical 
activities and games. 

The Glen Haven Training Program , developed by Dr. Newell Kephart and 
distributed by Learning Pathways, Inc., has been in use since 1969. 

Designed for teachers of 25% of pupils in grades K-6, and focused on learn- 
ing procedures and methodology and on perceptual-motor skills, the materials 
aim at giving appropriate professionals the training necessary to help 
learners to overcome their disabilities. The local program requires 
approximately 30 hours of training sessions for professionals using manuals 
and films. 
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The Greater Cleveland Social Science Program , developed by the 
Educational Research Council of America, has been in use since 1970. 

Designed for all social studies pupils in grades K-6, the materials . 
encourage pupils to analyze values in human societies. Materials include 
texts, guides, and audio-visual aids. 

Harvard Project Physics Materials were developed by Dr. F. James 
Rutherford and have been in use since 1969. Designed for all science 
students in grades 9 through 12, the materials present physics in a broad 
humanistic context with maximum flexibility with regard to context, emphasis, 
and teaching strategies in order to encourage a wider variety of students 
to learn the principles of physics. Materials Include texts, guides, labo- 
ratory equipment, and audio-visual aids. 

The Hawaii English Project Materials , developed by the Hawaii State 
Department of Education and the University of Hawaii, have been in use 
since 1970. Designed for all language students in grades K-6, the materials 
teach language and literature together and allow children to perfoinn accord- 
ing to individual talents and temperaments. Features of the program are 
Inquiry approaches, activity— centered learning, specific objectives, built- 
in evaluation, and peer teaching. Materials include texts, workbooks, 
audio-visual aids, displays, games, and tests. 



The Heath Urban Reading Program was developed by Educreative Systems, 
Inc., and has been in use since 1970. Designed for 25% of language students 
in grades 7 through 9, the textbooks aim at improving the reading skills of 
Intermediate students who read below their grade level. 

The High School Geography Project Materials , developed by the Associa- 
tion of American Geographers, have been in use since 1969. Designed as a 
one-year course for all social studies students in grades 9 through 12, the 
project attempts to develop student inquiry and decision-making attitudes 
and abilities. Materials include texts, manuals, workbooks, audio-visual 
aids, and teacher-preparation kits. 

Xhe H"man Relations Training Unit was developed by the Far West 
Laboratory for Educational Research and Development and has been in use 
since 1970. Designed for 75% of 9th through 12th grade students, the films, 
texts , and manuals attempt to reduce attitudinal and communication barriers 
among teenagers and adults— particularly between ethnic groups. Materials 
focus on learning procedures and methodology. 

The Idea-Centered Laboratory Science Project Materials , developed by 
W. C. Van Deventer of Western Michigan University, have been in use since 
1967. Designed for 50% of 7th and 8th grade science students, the program 
is idea-centered and activity-based, reduces required reading to a bare 
minimum, and is particularly directed at slow learners. The texts, tests, 
and self-explanatory teacher notes aim at helping students understand the 
scientist's point of view and the kinds of things he does«»«) 
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I Know A Place was developed by Robert Tannen and CSCS, Inc., and 
has been in use since 1969. Designed for all social studies pupils in 
grade 3, the texts attempt to increase the pupil's growth by exercising 
some of his developing skills and asking the pupil to organize and focus 
his knowledge and experiences in order to become a more effective com- 
municator of ideas. Texts and a teacher's booklet comprise the materials. 

Improving Motor— Perceptual Skills , developed by the Northwest Regional 
Educational Laboratory, has been in use since 1970. Designed for all 
kindergarten pupils, the teacher's guide assists teachers in providing 
practice and training to help children develop motor-perceptual abilities. 

Individualized Mathematics System , developed by the Regional Educa- 
tional Laboratory for the Carolinas and Virginia, has been in use since 
1970. Designed for all mathematics students in grades 1 through 6, the 
materials allow children to progress at individual rates in dictated se- 
quences dependent on group and individual tests and on different combina- 
tions of learning materials. The texts, tests, and manipulative devices 
provided aim at increasing pupil learning efficiency, motivation, and 
self-confidence . 

Individually Prescribed Instruction was developed by the Learning 
Research and Development Center of the University of Pittsburgh and has 
been in use since 1967. Designed for all mathematics pupils in grades 
K-6, the program uses instructional objectives and printed and audio- 
visual self-instructional materials. Kits and displays are also utilized. 

The Inquiry Development Program in Earth Science , developed by J. 
Richard Suchman and distributed by Science Research Associates, Inc., has 
been in use since 1968. Designed for 50% of 8th through 10th grade sci- 
ence students, the materials, including texts, manuals, kits, films, and 
workbooks, aim at stimulating students' self-directed theorizing and 
experimenting . 

The Inquiry Development Program in Physical . Science , developed by 
J. Richard Suchman and distributed by Science Research Associates, Inc., 
has been in use since 1966. Designed for all science students in grades 
6 through 8, the texts, manuals, kits, films, and workbooks aim at stimu- 
lating students' self-directed theorizing and experimenting. 

The Inquiry Role Approach Materials were developed by the Mid-Continent 
Regional Educational Laboratory and have been in use since 1968. Designed 
for 75% of 10th grade science students, the program organizes classes into 
teams of four in order to increase student participation, involvement, 
responsibility, and level of inquiry. It utilizes Biological Sciences 
Curriculum Study materials as well as manuals and audio-visual aids. 

The Instructional Concepts Program , developed by Southwest Regional 
Laboratory for Educational Research and Development, has been in use since 
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1970. Designed for all kindergarten pupils, the program focuses on the 
language subject area, teaching pupils concepts of reading readiness. 

Materials include guides, charts, and pupil booklets. 

Intergroup Relations Curriculum Materials , developed by the Lincoln 
FilenrCenter for Citizenship and Public Affairs of Tufts University, has 
been in operation since 1970. Designed for all social studies pupils in 
grades 1 through 6, the materials aim at developing student self-awareness 
and understanding of others. Texts, photographs, films, and manuals are used 
in the complete one-year program. 

Intermediate Science Curriculum Study Materials were developed by 
Florida State University and have been in use since 1970. Designed for 
all science students in grades 7 through 9, the program permits the pace 
and level of instruction to be adapted to the ability and interest of each 
student. Materials include laboratory kits, guides, manuals, behavioral 
objectives, and evaluation materials. 

The Interrelated Mathematics Science Project Materials , developed by 
Nova Schools, Secondary Division, Ft. Lauderdale, Florida, have been in 
use since 1969. Designed for all mathematics and science students in grades 
9 through 12, the individualized program of instruction strives to teach 
mathematics and science interrelatedly using learning activity packages. 

Introductory Physical Science , developed by Educational Development 
Center, has been in use since 1966. Designed for all science students in 
grades 8 and 9, the program strives to offer students insight into the 
means by which scientific knowledge is acquired, as well as to offer stu- 
dents an elementary knowledge of physical science. Materials include a 
text, teacher’s guide, laboratory equipment, tests, and films. 

The Language Master, developed by Bell and Howell, has been in use 
since 1965. Designed for use by all students, particularly in language and 
mathematics, the machine teaches pupils new vocabulary by allowing them to 
simultaneously see and hear new words or terms and to proceed at an individual 

pace. 

The Learning Booth, developed by Far West Laboratory for Educational 
Research and Development, has been in use since 1969. Designed for all 
language pupils in kindergarten and 1st grade, the program requires children 
to participate only when they wish to do so. A booth attendant responds 
to the child’s needs and desires rather than requesting the child to per- 
form. The booth consists of a special electric typewriter and related 
materials. The purposes of the booth are to increase the pupil s ability 
to solve problems, increase his language development, improve the use of his 
senses and perceptions, and develop his conceptual abilities. 

The Learning Mastery System was developed by the Southwest Regional 
Laboratory for Educational Research and Development and has been in use 
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since 1967. The program is designed to help all 1st grade language 
pupils using Harper and Row and Macmillan Bank Street reading programs 
to use those programs efficiently in learning to read. Teacher's manuals 
comprise the materials. 

The Listening-Reading Program was developed by Educreative Systems, 

Inc., and has been in use since 1970. Designed for all language pupils in 
grades 1 through 3, this supplementary reading program uses records and 
story pamphlets to develop the pupil's listening and reading skills. Pupils 
are motivated to read by being presented the first half of a story on 
record, and then they must read the second half. 

LLL (Listen Lock Learn) w as developed by Educational Developmental 
Laboratories, Inc., anJ has bean in use since 19bf’. Designed for 75% of 
all language students in grades K through 5, this program is ungraded and 
individual in its approach to teaching basic listening, reading, and speak- 
ing skills. Materials Include texts, films, tapes, and workbooks. 

Man: A Course of Study, developed by Education Development Center, 

has been in use since 1970. Designed for all social studies pupils in 
grades 5 through 7, the nine-month course developes Inquiry-based, inter- 
disciplincrv curricuLi T-rhirh focus on the unifying questions of man's 
experiences. n.ctorio i include manuals, guides, gamc.s, films, slides, 
journals, poetry, and songs. 

Man in Action. Ic^e loped by the Brentwood Public Schools and distributed 
by Prentice-Hall, lac., has been in use since 1968. Designed for all social 
studies pupils in grades K-6, the program is organized around key social 
science concepts and presented in a way that actively involves students in 
the learning process. Materials include student texts and teacher manuals. 

The course strives to teach children the key ideas of the social sciences 
so that they will be able to function more effectively in school and as 
adults. 

The M<an Made World, developed by Polytechnic Institute of Brooklyn, 
has been in use since 1969. Designed for 75% of 10th through 12th grade 
mathematics, science, and social studies students, the text is directed 
toward students who will .ittend college but do not plan to. study science 
or engineering. It is expected to "help students learn to live with and 
manage technological literacy and environmental quality." 

The Materials and Activities for Teachers and Children Program , developed 
hv the staff"of Children 'r, Mtisct'.m in Boston, has been in use since 1965. 
Desicn.-!d for all social studies pupils in grades K-6, the program strives 
to assist the pupil in developing a conceptual framework for better under- 
standing of various aspects of life; e.g., city life. The materials, includ- 
ing films, pictures, games, and recordings, are highly motivational and 
involve meaningful non-verbal tasks for the pupil . 
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Mathematics; Modern Concepts and Skills , developed by D.C. Heath 
& Co., have been In use since 1969. Designed for 25% of 7th through 9th 
grade mathematics students , materials are aimed at teaching slow learners 
basic mathematic skills and applications of mathematics In dally life, as 
well as Introducing basic concepts of geometry, number theory, and algebra. 
Three texts, teacher's editions, workbooks, tests, and audio— visual aids 
comprise the materials used. 

The Merrill Linguistic Reading Program , developed by Charles C. Fries 
of the Charles E, Merrill Publishing Company, has been In use since 1966. 
Designed for all language pupils In grades 1 through 3, the program utilizes 
a linguistic approach and represents complete beginning reading Instruction. 
Reading texts comprise the program materials. 

The Michigan Social Science Education Prolect Materials were developed 
by the University of Michigan and have been In use since 1968. Designed 
for all social studies pupils In grades 4 through 6, the program strives to 
help children use social science methodology In scientific Inquiry to examine 
problems of behavior, using themselves and the classroom as a laboratory. 
Materials, Including texts, records, and guides, makeup seven units, each 
requiring six to eight weeks' study. 

Mike's World— Your World , developed by Hill Development Corporation 
and Wesleyan University and distributed by Education Ventures, Inc., has 
been In use since 1969. Designed for all science pupils In grades 5 and 6, 
the program Includes readings, simulation games, and field work aimed at 
giving pupils Information and promoting their commitment In the area of 
environmental conservation. The program Is a supplement to the regular 
science curriculum. 

The Minnesota Mathematics and Science Teaching Project Materials were 
developed by the University of Minnesota and have been In use since 1966. 
Designed for all mathematics and science pupils In grades K-3, the program 
jfgpresents coordinated teaching of mathematics and science through a four— 
year "spiral" curriculum using a discovery approach with manipulative 
materials. Materials Include texts, films, games, and laboratory equipment. 

Movlgenlc Orientation, developed by Ray Barsch, has been In use since 
1967. Designed for all students at all grade levels, the program focuses 
on perceptual -motor Improvement In order to make students better Integrated 
learners. Texts comprise the materials used. 

The Multiunit Elementary School Model , developed by the Wisconsin 
Center for Research and Development, has been In use since 1968. Designed 
foj; all pupils In grades K— 8, the model replaces standard elementary class- 
rooms with Instruction and Research Units and Incorporates team teaching, 
continuous progress, flexible use of materials, ungradedness, and differ- 
entiated Instructional roles In order to Increase pupil learning and raise 
pupil and teacher morale* 



Mew Directions In English , developed by Educational Development 
Corporation of Palo Alto and distributed by Harror and Row, School 
Department, has been in use since 1968. Designed for 50% of language 
students in grades 1 through 8, the program uses a guided discovery 
approach for students of average and above average ability and is lin- 
guistically oriented. Texts and workbooks comprise the materials pro- 
duced, which represent a complete program to teach language and composition 

skills. 

nff -Farr Agricultural Occupational Materials , developed by the Center 
fer H^oirch and Leadership Development in Vocational and Technical Educa- 
tion, have been in use since 1966. Designed for 50% of vocational education 
in grades 11 and 12, the program provides course outlines and stu- 
dent materials in module form for several off-T ,rm agricultural occupation 
courses. 

The Open Court Correlated Language Arts Program was developed by the 
Open Court Publishing Company, and has been in use since 1965. Designed 
for all language pupils in grades 1 through 6, the program is strongly 
teacher-directed and strives to teach children to read independently by 
the end of first grade and attempts to provide selections of literary 
quality and rewarding content. Texts and manuals form tlic core of the 
progr.am. 



Operation Co mpu/ Tel was developed by the Illinois Institute of Tech- 
nology and r.as been in tise since 1966. Designed for 25% of 8th through 
12th grade mnfhem.stirp sttidcnts, the programmed text and computer hardware 
aid in reaching students to write computer programs using IITRAN and to 
I" r;a:.hcmatics problems with the programs. 

The Palo Alto Reading Program , developed by the Palo Alto Unified 
School District and distributed by Har court Brace Jovanovich, Inc., has 
been in use since 1967. Designed for all language pupils in grades 1 
through 3, the program represents an eclectic (basal and linguistic) 
approach to teaching reading skills and strives to make step-by-step pro- 
gress possible for all pupils in beginning reading and writing. Materials 
Include texts, workpads, guides, and kits. 

Patterns and Processes of Science was developed by Weisbrack, Brock, 
and Paulsen and is distributed oy D.C. Heath and Company. The materials 

h. ivo been in use since 1968 and are designed for 75% of 7th through 9th 
grade science students. Laboratory textbooks enable students to develop 

i. nSov^tory skills, perform and .study experiments using scientific processes, 
nn 1 dlsccvpr patterns In the universe. 

rir;.crns in Arithmetic , developed by the Wisconsin Research and 
Development Center for Cognitive Learning, has been in use since 1967. 
Designed for all mathematics pupils in grades 1 through 6, the program 
utilizes a spiral sequence with behavioral objectives and televised 
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instruction of key concepts in new mathematics. Materials include tele- 
vision presentation, workbooks, and teacher's manuals. Forty-five minutes 
daily participation is recommended. 

People and Their Environment , developed by the J.G. Ferguson Publishing 
Company, has been in use since 1969. Designed for all students in all leve s 
(K-12) of science and social studies, the program can be used as a complete 
or supplementary environmental studies curriculum. Its purpose is to acquaint 
students with nature and to teach them wise environmental management. Teacher 
manuals provide detailed lessons that attempt to integrate conservation in 
all subject areas. 

Phvsical Science II was developed by Education Development Center and 
has been in use since“l970. Designed for 75% of science students in grades 
9 through 12, the program extends the purpose of the Introductory Physical 
Science course (see above). Materials include texts, manuals, tests, and 
laboratory equipment and are available only to teachers trained in an 
approved institute or workshop. 

Progra mmed Tutoring Reading , developed by Douglas Ellson of the University 
of Indiana, has been in operation since 1968. Programmed materials are aimed 
at improving the reading skills for 25% of first grade language pupils. 

Project I nsight: Human Relations Curriculum was developed by the PACE 

Association and has operated since 1966. Aimed It all social studies students 
in grades K-12, the program utilizes an open classroom and affective learning 
techniques. It can be used as a complete one-year program or as supplementary 
units, intended to help students and teacher respond to critical human rela- 
tions’ incidents. Films, role-playing, and games are used. 

Project Local (Laboratory Program for Computer-Assist ed Learning), 
developed by Robert N. Haven ^d the Town of Westwood, Massachusetts, has 
been in use since 1969. Designed for 75% of 9th through 12th graders in 
mathematics, science, anl social studies, the program strives to integrate 
use of the computer as a teaching aid for problem solving, class demonstra- 
tion, and computer-based drill and practice into the regular curricula. 
Materials include a programmed text, teacher's manual, and resource materials. 

Project PLAN (Program for LeaminR in Accordan ce with Needs) , developed 
by John C. Flanagan and the American Institutes for Research and distributed 
by Westinghouse Learning Corporation, has been in operation since 1967. 
Designed for all grade 1 through 12 students in language, mathematics, 
science, social studies, and guidance, the program provides computer-imaged 
individualized instruction as a complete curriculum or as separate subject 
area courses. Provided are computer support, teacher-training materials, 
teaching-learning units, tests, and recommendations for all necessary instruc- 
tional materials. 
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Project Social Studies Materials , developed by Edith West of the 
Minnesota Curriculum Development Center, University of Minnesota and dis- 
tributed by Green Printing Company, have been in use since 1970. Designed 
for all social studies students in grades K-12, the curriculum materials 
place increased emphasis on the behavioral sciences and the non-western 
world and aim at helping students develop inquiry skills as well as a sound 
knowledge of the social sciences. Materials include teacher guides and 
resource units. 

The Providence Social Studies Curriculum Project Materials were 
developed by R.F. Skinn, Jr. of the Providence, Rhode Island Public Schools 
and are distributed by the Rhode Island College Bookstore. The program, in 
use since 1967, is designed for all social studies students in grades K-12 
and strives to increase student activity and reduce traditional textbook 
reading, recitation, and testing in the learning of social studies skills 
and content. Emphasis is placed on individual and small group projects and 
student self-direction. Materials provided include curriculum guides and 
resource units; schools must purchase texts and pamphlets for the full 13- 
year course. 

The Quantitative Approach to Elementary School Science , developed by 
Clifford E. Swartz of the State University of New York, Stony Brook, has 
been in use since 1970. Designed for all science pupils in grades 3 through 
6, the program combines mathematics and science and treats only phenomena 
and concepts which can be seen or manipulated by the child in the teaching 
of elementary science. Materials include texts, kits, and teachers' guides. 

Quantitative Physical Science was developed by Dr. Sherwood Githens, Jr. 
of Duke University and has been in use since 1967. Designed for all 9th 
grade science students, the course is an equipment-based learning program 
consisting of a series of 90 manipulative learning operations. The program 
serves as a transition between elementary and secondary science courses and 
strives to teach the student the fundamentals of instrumentation, measure- 
ment, and various scientific principles. A text and teacher's manual supple- 
ment laboratory equipment. 

The Random Access Data Storage and Retrieval System , developed by 
Bruce R. Joyce of Columbia University Teachers College, has been in use 
since 1968. Designed for all social studies pupils in grades 2 through 
6, the system gives pupils access to a large amount of information on the 
topic of Pueblo culture to enable them to study the culture easily and 
thoroughly. Materials include thousands of modules of information, tapes, 
slides, pictures, legends, dramatizations, and displays. 

The Reading Caravan was developed by D.C. Heath and Company and has 
been in use since 1968. The program is designed for 75% of 1st through 
6th grade language pupils. This seven-book supplementary reading series 
with accompanying records are aimed at developing pupils' basic reading 
skills . 
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The Responsive Environment Model , developed by Glen Nimnicht of the 
Far West Laboratory for Educational Research and Development, has been in 
operation since 1968. Designed for all grade K-3 pupils in language, mathe- 
matics, science, and social studies, the model represents a complete program 
for early childhood and psycho-social development. Using a self-directed 
approach, the program strives to allow the child to explore and make dis- 
coveries of his own. Materials include texts, audio-visual aids, Kits, 
games, and displays. Local school advisors require two weeks of training. 



Scie nce; A Process Approach was developed by the American Association 
for the Advancement of Science Coimnission on Science Education a^ has been 
in use since 1967. Designed for all science pupils in grades K-6, this is 
a complete program to be used in a designated sequence, stressing the intel- 
lectual processes of science organized as hierarchies of skills and subskillj. 
Student performance is evaluated by means of behavioral objectives. No 
reading is involved; lessons progress through demonstrations, activities, 
discussions, and evaluation. 



Th e Science Curriculum Improvement Study Materials , developed by the 
University of California at Berkeley, have been in use since 1968. Designed 
for all science pupils in grades K-6, the materials may be used as a complete 
science curriculum or as supplementary units. Life and Physical science con- 
cepts are developed through direct "discovery experiences led by the teacher. 
One and one-half hours of weekly instruction are recommended. Manuals, 
laboratory equipment, kits, and workbooks comprise the materials. 



The Secondary School Mathematics Curriculum Improv ement Study Materials, 
developed by Columbia University Teachers College, have been in use since 
1968. Designed for 25% of grade 9 through 12 mathematics students, the 
program aims at university-bound students and includes large amounts of 
current university study. Textbooks comprise the means of instruction. 



The second ary School Science Project Materials were developed by 
Rutgers University and have been in use since 1966. Designed for all science 
stuLnts in grades 8 through 10, the program emphasizes what tte stu^nt hM- 
self is able to do given access to primary sources of information, ^e course 
is designed to help the student reach some understanding of the physical 
world and to experience firsthand the investigative nature of science. 

Student Investigation Books and paperbacks take the place of standard texts. 



A Self-Instructional System in Basic Electricity , developed by the 
Northwest Regional EducaUonal Laboratory, has been in use since 1968. 
Designed for 25% of vocational education students in grades 7 through IZ, 
the program strives to assist students to acquire fundamental concepts of 
electricity and how they are applied in daily life. Progr^ed teaching 
techniqites and devices permit each student to proceed at his own pace for 
the average four-week, 12-hour course. Materials include audio-visual 
aids, workbooks, test, guides, and equipment. 



A Self-Instructional System In Speech , developed by the Northwest 
Regional Educational Laboratory, has been In use since 1968. Designed for 
75% of language students In grades 7 through 12, the program strives to 
assist students In acquiring the basic skills In public speaking. The 
system can be Incorporated Into a beginning speech course or used by Indi- 
vidual students. Materials Include films and programmed units of Instruc- 
tion. 



A Self-Instructional System In Welding was developed by the Northwest 
Regional Educational Laboratory and has been In use since 1968. Designed 
for 25% of vocational education students In grades 9 through 12, the pro- 
gram strives to provide students with the basic skills of electric arc 
welding through self-lnstructlon. The system can be used as a self-contained 
unit or combined with an existing course. The program uses a variety of media. 
Including printed, audio and visual materials and variable step sizes In 
the learning sequence. The teacher /manager need not be highly skilled In 
welding. 

The Self-Understanding Through Occupational Exploration Program , 
developed by the Oregon State Department of Education, has been In use since 
1967. Designed for all vocational education students In grades 8 and 9, the 
goal of the program Is to enable students to examine their own capabilities 
and Interests and many career possibilities In order to make tentative career 
cluster decisions and to design their own educational and training programs. 
The teacher's guide provides complete recommendations for materials to be 
used. 

Sesame Street was developed by Children's Television Workshop and has 
been on the air since 1970. Designed for all pupils In language and mathe- 
matics In kindergarten and first grade, the dally, hour-long color television 
series strives to enhance children's cognitive skills by means of frequent 
repitition of spot "commercials" and commercial TV entertainment techniques. 
Books and records are also produced. 

Social Sciences; Concepts and Values , developed by Paul Brandwein of 
Har court, Brace, & World, has been in use since 1970. Designed for all 
social studies pupils in grades K-6, the program presents a sequential 
series of materials organized around key social science concepts and values. 
Student and teacher texts comprise the materials. 

The Sullivan Reading Program , developed by M.W. Sullivan And distri- 
buted by Behavioral Research Laboratories, has been in use since 1967. 

Designed for all levels of language students who require improved reading 
skills, the program uses a programmed, linguistic approach with emphasis 
on decoding skills. Materials include texts, tests, and displays. 

The Taba Curriculum Development Project Materials , developed by San 
Francisco State College, have been in use since 1969. Designed for all 
social studies pupils in grades 1 through 8, the program strives to develop 
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students' thinking skills, allow acq'ulsltlon of selected knowledge, and 
form selected attitudes. Materials consist of a series of curriculum 
guides In the form of teaching-learning units. 

Taking a Stand (Oliver Materials) was developed by the Harvard Social 
Studies Project and Is distributed by American Education Publications. It 
has been In use since 1967. Designed for 50% of 7th through 9th grade social 
studies students , the program uses critical thinking, social science, law, 
and philosophical concepts to teach students to develop opinions on contro- 
versial public Issues. Films and pamphlets comprise the materials. 

The Talking Typewriter , developed by Omar K. Moore atid Responsive 
Environments Corporation, has been In use since 1965. Designed for 75% 
of language pupils In grades K-6, the program allows children to proceed 
with reading skills at their own pace, encourages participation, and 
elicits responses In the non- threatening atmosphere of a private booth. 
Materials Include a special typewriter keyboard and other computer com- 
ponents. 

The Technology for Children Project Materials , developed by Fred J. 
Dreves and the New Jersey State Department of Education, Vocational Division, 
has been In use since 1968. Designed for all kindergarten- 3rd grade pupils, 
the program strives to enlarge the child's understanding of vocational .lolce 
and to develop his economic competence In a changing world of work. 

Total Education In the Total Environment Materials were developed by 
William R. Eblen of the Wilton, Connecticut Public Schools and have been 
In use since 1968. Designed for all mathematics, science, and social studies 
students In grades K-12, the program strives to enable students to develop an 
appreciation of their Interdependence with their total environment and to 
encourage responsibility for maintaining a livable environment. Materials 
Include manuals, workbooks, and audio-visual aids. 

Total Reading , developed by Mary Johnston and Ruth Scanlon of Carmel, 
California, has been In use since 1965. Designed for all language pupils 
In grades 1 through 3, the texts Integrate the teaching of reading, writing, 
spelling, and speech through an understanding of the phonetic components of 
English. Pupils work Independently at early stages. Texts and tests com- 
prise the materials. 

UNIPAC was developed by the Kettering Foundation's Institute for 
Development of Educational Activities and has been In use since 1968. 

Designed for all pupils In all areas at all levels, materials consist of 
Individualized learning packages written by teachers In all parts of the 
country. 

The University of Il linois Astronomy Program , developed by James Atkin 
of the University of Illinois and distributed by Harper and Row, School 
Department, the program has been In use since 1970. Designed for 75% of 




D-18 

164 



science students in grades 5 through 10, the program can be used as six 
study units to supplement a regular course or as a complete one-year course. 
The units employ a "story-line" approach, unfolding the history of astronomy; 
of scientists acquiring understanding of the solar system, stars, and 
galaxies; behavior of light; and gravitation. 

Variable Modular Scheduling via Computer , developed by Stanford 
University and Educational Coordinates, has been xn use since 1967. Designed 
for all pupils in all areas and at all levels, the program allows students to 
schedule flexible class periods and work within a curriculum removed from the 
rigid confines of six standard class periods per day. 

Vocational Information for Education and Work , developed by Appalachia 
Educational Laboratory, Inc., has been in use since 1967. Designed for 75/^ 
of vocational education students in grades 9 through 12, the materials sup- 
ply students with occupational information on jobs requiring less than a 
college degree for which training is available regionally and/or there is 
an accelerating need for workers. Materials include microfiche scripts. 



The World of Language, developed by Follett Educational Corporation^ 
and in use since 1970° was designed for all language pupils in grades K-8. 

A complete program in language arts, materials emphasize the oral aspects 
of language and aim at increasing pupils' skills relating to communication, 
language, cognitive development, human relations, and literature. Materials 
include poems, stories, art work, and photographs. 



APPENDIX E 



DETAILED DESCRIPTION OF THE 117 SELECTED PRODUCTS 



The following codes are used in the table presented below that 
describes each of the 117 selected products. See the text for further 
definitions of the data items. 



1. Focus 

SM. 

0 . 

CO. 

LPM. 

PM. 

2. Content 

L. 

M. 

S. 

SS. 

VE. 

3. Format 

TB. 

W. 

M. 

TV. 

F. 

A. 

D. 

G • 

K. 
C. 

L. 

T. 



Subject Matter Content 
Organizational Efficiency 
Classroom Climate or Operation 
Learning Procedures or Methodology 
Perceptual— Motor Skills 



Language 

Mathematics 

Science 

Social Studies 
Vocational Education 



Textbooks (teacher and student editions) 

Workbooks (including laboratory books, supplementary worksheets) 
Manuals, guides 
Television series 

Films (includes movies, filmloops, filmstrips, slides) 

Audio equipment (includes records, tapes, cassettes) 

Displays (includes charts, graphs, models, maps, globes, 
atlas, flash cards) 

Games 

Kits 

Computer hardware and software 

Laboratory equipment 

Tests 
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4. Grade Level 



Grade level K (kindergarten) through 12 is shown. 

5. Percent of Target Population 

Coded as 25%, 50%, 75%, or 100% 

6. Number of Schools 

The actual number of schools using the product is noted. 

7. Number of Students 



Number of students using the product is noted. 

8. Degree of Dissemination 

H. High degree of dissemination (i.e. , product brought to the 
attention of the educational profession through more than 
10 known sources) . 

M. Moderate degree of dissemination (i.e., 6-9 sources). 

L. Low degree of dissemination (i.e., 2-5 sources). 

9. Year Came Into Use 

The year (i.e., 1965-1970) the product was first adopted is 
noted. 

10. Measured Cognitive Gains — Nominator 

Y. Yes 
N. No 

DK. Don't Know 

11. Measured Cognitive Gains — Evidence 

For each product, one (the most positive one) of the following 
letter codes was used: 

PP-ST-C. Pre- and post standardized test measures with control 
groups . 

PP-ST. Pre- and post standardized test measures without 
control groups. 

PP-PT-C. Pre- and post product test measures (i.e., criterion 
tests or other measures or indicators of performance 
specified as evidence of attainment of the product 
objectives) with control groups. 
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PP-PT. 



P-PT-C . 
P-PT. 
UNSP. 



NE. 



DK. 



Pre- and post product test measures without control 
groups . 

Post only product test measures with control groups. 
Post only product test measures without control groups. 
Unspecified gain — gain not hypothesized by evaluator. 
Negative evidence on effectiveness of product. 

Don't Know — no evidence found by staff. 



12 . 



13. 



14. 




Sustained Cognitive Gains — Evidence 



For each product, one (the most positive one) of the following letter 
codes was used: 




RMS. 


Repeated (i.e., two or more tests or measures) measures 
vin standardized tests. 




PPS. 


Pre- and post-measures via standardized tests. 




RMP. 


Repeated measures of product tests. 




PPP. 


Pre- and post-measures via product (criteria) tests. 




P. 


Post only measure. 




DK. 


Don ' t Know . 


Measured Affective Gains — Nominator 




Y. 


Yes 




N. 


No 




DK. 


Don't Know 


Measured Affective Gains — Evidence 


For each product, one (the most positive one) of the following letter 
codes was used: 




PP-PT-C. 


Pre- and post-measures via product test with control 






groups . 




PP-PT. 


Pre- and post-measures via product test without 
control groups. 




P-PT-C . 


Post-only measures via product (criterion) test with 
control group. 




P-PT. 


Post-only measures via product test without control 






groups . 




UNSP. 


Unspecified gain — gain not expected or hypothesized. 




NE. 


Negative evidence on effectiveness of product. 




DK. 


Don't Know. 
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15. Sustained Affective Gains — Evidence 



For each product, one (the most positive one) of the following letter 
codes was used : 



RMP. 


Repeated (l.e., two or 
of the product tests. 


more tests or measures) measures 


PPP. 


Pre- and post-measures 


via product test. 


P. 


Post-only measure. 




DK. 


Don't Know. 





16. Contributed Toward Gains — Nominator 



"Don't Know" was marked if the nominator indicated "Don't Know" for 
all of the areas; otherwise, "Yes" or "No" was indicated for each area 
for which the nominator marked "Yes" or "No." The following are the 
relevant code letters: 

OY. Marked Yes to Organizational Efficiency. 

ON. Marked No to Organizational Efficiency. 

ODK. Marked Don't Know to Organizational Efficiency. 

COY. Marked Yes to Classroom Climate or Operation. 

CON. Marked No to Classroom Climate or Operation. 

CODK. Marked Don't Know to Classroom Climate or Operation. 

LPMY. Marked Yes to Learning Procedures or Methodology. 

LPMN. Marked No to Learning Procedures or Methodology. 

LPMDK. Marked Don't Know to Learning Procedures or Methodology. 

PMY. Marked Yes to Perceptual-Motor Skills. 

PMN. Marked No to Perceptual-Motor Skills. 

PMDK. Marked Don't Know to Perceptual-Motor Skills. 

17. Contributed Toward Gains — Evidence 



"Don't Know" was marked if no supporting evidence was available for all 
four areas; otherwise, "Yes" or "No" was indicated for each area for 
which additional Information was available. The following are the 
relevant code letters: 

OY. Organizational Efficiency; Yes, evidence found. 

ON. Organizational Efficiency; No, evidence not found. 

ODK. Organizational Efficiency; Don't Know. 

COY. Classroom Climate or Operation; Yes, evidence found. 

CON. Classroom Climate or Operation; No, evidence not found. 

CODK. Classroom Climate or Operation; Don't Know. 
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LPMY. Learning Procedures or Methodology; Yes, evidence found. 

LPMN. Learning Procedures or Methodology; No, evidence not 
found . 

LPMDK. Learning Procedures or Methodology; Don't Know. 

PMY. Perceptual-Motor Skills; Yes, evidence found. 

PMN. Perceptual-Motor Skills; No, evidence not found. 

PMDK. Perceptual-Motor Skills; Don't Know. 

18. Developer 

The type of agency was coded as follows; 

G. Government (including R&D laboratories and centers, 
state universities, state departments, etc.) 

PNP. Private Non-Profit (e.g., AIR, ETS, Ford Foundation). 

PPM. Private Profit-Making (e.g., publishers, individual 
enterprises) . 

Id. Location of Developer 

NE. Northeast (includes USOE Regions 1, 2, and 5; 

Connecticut, Delaware, Illinois, Indiana, Massachusetts, 
Michigan, New Hampshire, New Jersey, New York, Ohio, 
Pennsylvania, Puerto Rico, Rhode Island, Vermont, 
Wisconsin) . 

SE. Southeast (includes USOE Regions 3 and 4: Alabama, 

Florida, Georgia, Kentucky, Maryland, Mississippi, 
North Carolina, South Carolina, Tennessee, Virginia, 
Virgin Islands, West Virginia). 

SC. South Central (includes USOE Region 7; Arkansas, 
Louisiana, New Mexico, Oklahoma, Texas). 

NC. North Central (includes USOE Regions 6 and 8; 

Colorado, Idaho, Iowa, Kansas, Minnesota, Missouri, 
Montana, Nebraska, North Dakota, South Dakota, Utah, 
Wyoming) . 

W. West (includes USOE Regions 9 and 10: Alaska, 

Arizona, California* Guam, Hawaii, Nevada, Oregon, 
American Samoa, Trust Territory of the Pacific, 
Washington) . 
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APPENDIX F 



MASTER OUTLINE FOR PRODUCT DEVELOPMENT REPORTS 



Explanation; The headings in the following outline are APA Order 
Headings. The following is an example of each of the four headings 
used : 



ORIGIN 

Key Personnel 



Special Qualifications 

Relevant experience . 



Note: First order heading is centered and in caps; second order heading 

is centered and underlined; third order heading begins at the left margin 
and is underlined; and fourth order heading is indented, underlined and 
followed first by a period (.) and then by the text of the paragraph. 

Words in "( )" are not to be included as part of the headings. When words 
appear in "[ select the appropriate word as part of the heading. 
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PRODUCT DESCRIPTION 
Product Characteristics 

Name 

Developer 

Distributor 

Focus 

Grade Level 
Target Population 

Rationale for Product 
Long Range Goals of Product 
Ob.lectives of Product 
Philosophy Behind Product 
Theories Supporting Product 

Description of Materials 

Organization of Materials 

Format of Materials (how physically presented) 

Content of Materials (concepts and terminology covered, etc.) 

Cost of Materials to User 

Procedures for Using Product 

Learner Activities 

Relationship to program objectives . 

Typical activities in a day . 

Group and individual activities. 
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Kinds of practice, review and feedljack . 



Recommended period of use . 

Provisions for mo'biva'bing student . 

Teacher Activities 

Teacher strategy . 

Teacher training . 

Out -of -class preparation . 

Provisions for Parent /Community Involvement 
Special Physical. Facilities or Equipment 

Rec«->TT»tnended Assessment Techniques for Users (e.g., criterion-referenced, 
tests, etc.) 

ORIGINS 

Key Personnel 

Education and Experience of Key Personnel 
Philosophy of Key Personnel 

Relevant Research Conducted by Key Personnel 

Sources of Ideas for Product 

Trends of the Time 
Relevant Research 

Theory . ^ 

Techniques . 

Technological Prerequisites 
Similar Products 

Evolution of Ideas for Product 

First Formulation of Ideas for Product (Note when, by whom, why, how, 
in what form?) 
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Chanf^es in Ideas for Product (Note when, by whom, why, how, in what form?) 
Factors Which Stimvilated Development of the Idea 
Efforts of a key person or persons . 

Available fxinding . 

Ne ed for the product . (Note how need was defined, assessed and 
documented. ) 

Potential effectiveness and feasibility of product . 

Motivation to produce product . 

Funding for Product 
Initial Efforts to Fund Products 

Contacts with Funding Sources 
Factors Influencing Funding Sources 
Preparation of Proposals 
Details of Funding Agreement 
Description of F\mding Sources 

Breakdown of Funds (by stages of development, categories of use, and/or 
components) 



PRODUCT DEVELOPMENT 
Management and Organization 

Characteristics of Development Agency (Note major funding source, age, 
other projects, n\jmber and qualifications of staff, organizational 
structure) 

Relationship of Thi s Product to Agency (Note proportion of resources, 
people and facilities devoted to product) 

Other Agencies Involved in Development 

Characteristics of other involved agencies. 



Relationship to primary developer . (Note division of responsibility, 
channels of communication, procedures for decision making.) 



Original Develo-pment Plan 



Ob.i actives 

Description of Expected Product 
Procedures for Product Development 
Organization . 

Tasks . 

Personnel . 

Time schedule . 

Planned Procedures for Product Evaluation 
Formative evaluation plans . 

Summative evaluation plans . 

Modifications of Original Development Plan 
List and Description of Modifications 
Reasons for Making Modifications 

Brief Comparison of Planned Development with Actual Development 

Actual Procedures for Development of Product 
(For each stage /phase of development note the 
following information) 

Development Staff (Note size, education, experience, special 
qualifications, organizational structure, problems in recruit- 
ing and maintaining.) 

Patterns of Interaction (Note channels of communication between 
staff, general interpersonal relationships . ) 

Development 

Activities and tasks . 

Procedures followed . (Note effective and not so effective ones.) 
Bottlenecks and problems . 

Major decisions . (Note procedures for making decisions.) 
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Formative Evaluation 



Conditions of formative evaluation . (Note when, by whom, 
with whom. ) 

Procedures followed in formative evaluation . 

Techniques used to gather information . 

Procedures for modifying product on the basis of formative 
evaluation results . 

Nature and extent of modifications . 

Number and description of formative evaluation cycles . 

Other Formative Evaluation (Note field tests, for example) 

Conditions of evaluation . (Note when, by whom, with whom.) 
Procedures followed . 

Techniques used to gather information . 

Procedures for modifying product on the basis of 
evaluation results . 

Nature and extent of modifications . 

Number and description of iterative cycles . 



Development of Performance Measures/Assessment Techniques 
(For each stage /phase of development note 
the following types of information) 

Development Staff (Note size, education, experience, special 
qualifications, organizational structure, problems in recruit- 
ing and maintaining) 

Patterns of Interaction (Note channels of communication between 
staff, general interpersonal relationships) 

Development 

Activities and tasks. (Note research on available techniques, 
adaption of techniques to product, expansion of techniques. 

Procedures followed. (Note effective and not so effective ones.) 
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Bot t lenecks and problems. 



( 



Ma,1or decisions . (Note procedvires for making decisions.) 

Formative Evaluation of Assessment Techniques 

Conditions of evaluation . (Note when, by whom, with whom.) 
Procedures followed . 

Techniques used for gathering information . 

Procedures for modifying assessment techniques . 

Nature and extent of modifications . 

Number and description of evaluation cycles . 

Other Formative Evaluation of Assessment Techniques ( e . g . , note 
field tests, etc.) 

Conditions of evaluation . (Note when, by whom, with whom.) 
Procedures followed . 

Techniques used for gathering information . 

Procedures for modifying assessment techniques . 

Nature and extent of modifications . 

Number and description of evaluation cycles, . 

SUMMATIVE EVALUATION 

Evaluation Staff 

Relationship to Development Staff 
Size 

Education, Experience, Special Qualifications 

Hierarchy and Organizational Structure (Note interpersonal relationships) 
Problems in Recruiting and Maintaining Staff 
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Field Tests 

(For each field test note the following 
types of information) 

Designer of Field Test (Note who. and when) 

Funding 

Coordinator of Field Test (Note who, relationship to developing 
and funding agencies) 

Sub.jects 

UnTti'hpy [students, schools, classes or teachers]. 
Geographical distribution . 

Socio-economic description . 

Selection process . (Note schools or districts and 
experimental and control groups) 

Treatments 



Experimental treatments . 
Control treatment . 



Measures 

Description of measures . (Note standardized tests, staff 
constructed tests, questionnaires, structured observations, 
school visits, etc.) 

Rationale for measures employed . 

Procedures for administration . 

Results of Field Test 

Analyses used . 

Rationale for analyses . 

Student cognitive changes . 

Student . affective changes . 

Changes in facilitating factors . 
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Unanticipated changes . 



Documentation and reporting of results . 

Modifications Made in Product 

Procedures for modifying product on the basj^s 
"Afield test results . 

Nature and extent of modifications in product. 



Other Svunmative Evaluations 
(e.g., those conducted by the users) 

Designer of Evaluation Program (Note who and when) 

Funding 

Coordinator (Note who, relationship to developing and funding 
agencies) 

Subjects 

Number [students, schools, classes or teachers]. 
Geographical distribution * 

Socio-economic description * 

Selection process * (Note schools or districts and 
experimental and control groups.) 

Treatments 



Experimental treatments . 

Control treatments . 

Measures 

Description of measures . (Note standardized tests, 
staff constructed tests, questionnaires, structured 
observations, school visits, etc.) 

Rationale for measures employed . 

Procedures for admi n i strat ion . 



Results of Evaluation 



Analyses used . 

Rationale for analyses . 

Student cognitjye changes . 

Student affect jye changes . 

Changes in facilitating factors . 

Unanticipated changes . 

PoenTnpntation and reporting of result s. 

Modifications Made in Product as a Result of Evaluation Results 
Comnients on the adequacy of the eyaluations # 

DIFFUSION 

Agency Participation 
Agencies Inyolyed (Note characteristics) 

Relationships AmoPK Agencies 

Diffusion Actiyities of Each Agency (Give brief descriptions) 

Diffusion Strategy 
Developer of Plans /Strategy 
Outline of Strategy 
Target . 

Techniques for reaching target . 

Actual Diffusion Efforts 
Actiyities (list, describe, when, by whom) 

Responses to Diffusion Efforts 



Indications of interest . 



Early users . 

Revisions in Diffusion Strategy (Describe and give rationale 
for making revisions) 

Product Characteristics and Other Factors 
Affecting Diffusion 

Complexity of Product 
DivisiTjjlity of Product 

Compatibility of Product vith Other School Practices 
T eacher Training Required 

Ease vith Which Product Can be Communicated 
Comparison vith Other Products 
Economic Conditions and Attitudes of the Times 
Cost of Product 

Start-up costs . 

Continuation costs . 

Alternative products. 



ADOPTION 

Extent of Product Use 

(Differentiate between field tests, try outs and adopt 

Location of Users (Note geographic distribution) 

NuTtiTjer of Users (Note how many students, schools; number of 
copies sold, etc.) 

Socio-Economic Characteristics of Users 
Length of Time in Use 
Initiation of Adoption 
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Relationship Betveen User and Developer and/or Distributor 



Installation Procedures 
Necessary Physical Arrangements or Equipment 
Necessary Classrooxn Organization 
Importance of Teacher Training 

Availability of training programs . 

Development of training programs . 

Description of training programs . (Note length, expense, 
organization, mat trials, techniques used.) 

Evaluation of effectiveness of training . 

Extra Staff Requirements 

Supervisory requirements . 

Paraprofessional requirements . 

Consultants . 

Extent of Product Modification Possible 
Degree of Administrative Support Needed 
Importance of Public Relations Effort Prior to. Adoption 

Success of Installation Procedures 
Favorable and Unfavorable Conditions 
Effective and Not So Effective Procedures, 

Methods for Obtaining Feedback From Users 

Available Information From Users 

COSTS 

Total Cost of Product From Origin Through Adoption 
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Available Breakdovn of Costs 
(By stages of development, product components, personnel) 

FUTURE OF THE PRODUCT 

Expected Use or Impact of Product 
Anticipated Revisions of Product 

CRITICAL DECISIONS 

REFERENCES 

APPENDICES 
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APPENDIX H 



LIST OF PRODUCT DEVELOPMENT REPORTS PREPARED SEPARATELY 

The following Product Development Reports have been distributed 
separately. Copies may be obtained through ERIC or from the American 
Institutes for Research, Palo Alto, California. 

No. 1: First Year Communication Skills Program, 62 pages 

No. 2: Hawaii English Program, 81 pages 

No. 3: Intermediate Science Curriculum Study, 47 pages 

No. 4: Science Curriculum Improvement Study, 48 pages 

No. 5: The Sullivan Reading Program, 70 pages 

No. 6: The Creative Learning Group Drug Education Program, 52 pages 

No. 7: The Frostig Program for Perceptual-Motor Development, 64 pages 

No. 8: Science — A Process Approach, 52 pages 

No. 9: Variable Modular Scheduling Via Computer, 96 pages 

No. 10: Sesame Street, 51 pages 

No. 11: Arithmetic Proficiency Training Program, 49 pages 

No. 12: The Edison Responsive Environment Learning System or 
The Talking Typewriter, 33 pages 

No. 13: Holt Social Studies Curriculum, 66 pages 

No. 14: Distar Instructional System, 69 pages 

No. 15: Materials and Activities for Teachers and Children — 

The MATCH Program, 59 pages 

No. 16: Developmental Economic Education Program (DEEP) , 52 pages 

No. 17: Individually Prescribed Instruction — Mathematics (IPI-Math), 48 pages 

No. 18: The Cluster Concept Program, 69 pages 

No. 19: The Taba Social Studies Curriculum, 79 pages 

No. 20: Facilitating Inquiry in the Classroom, 38 pages 

No. 21: Program for Learning in Accordance With Needs (PLAN), 84 pages 
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